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ABOUT THE Al NOW INSTITUTE 


The Al Now Institute at New York University ¡s an interdisciplinary research institute dedicated to 
understanding the social implications of Al technologies. It is the first university research center 
focused specifically on Al's social significance. Founded by Kate Crawford and Meredith Whittaker 
in 2017, Al Now is one of the few women-led Al institutes in the world. 

Al Now works with a broad coalition of stakeholders, including academic researchers, industry, 
civil society, policymakers, and impacted communities, to understand and address issues raised 
by the rapid introduction of Al across core social domains. Al Now produces interdisciplinary 
research to help ensure that Al systems are accountable to the communities and contexts they 
are meant to serve, and that they are applied in ways that promote justice and equity. The 
Institutos current research agenda focuses on four core areas: bias and inclusión, rights and 
liberties, labor and automation, and safety and critical infrastructure. 

Our most recent publications inelude: 

• Dirty Data. Bad Predictions: How Civil Riahts Violations Impact Pólice Data. Predictive 

Polícínq Systems, and Justice. an article on how "dirty-policing" practices and policies 
shape the environment and the methodology by which data is created, raising the risk of 
creating inaccurate, skewed, or systematically biased "dirty data." 

• Anatomy of an Al System, a large-scale map and longform essay produced in partnership 
with SHARE Lab, which investigates the human labor, data, and planetary resources 
required to opérate an Amazon Echo. 

• Discriminatinq Systems: Gender, Race, and Power in Al. a report that examines how 
discrimination and inequality in the Al sector are replicated in Al technology and offers 
recommendations for change. 

• Disability, Bias. and Al, drawing on a wealth of research from disability advocates and 
scholars, this report examines what disability studies and activism can tell us about the 
risks and possibilities of Al. 

• Excavatina Al . an essay on the politics of images in machine learning training sets. 

• Litiqatinq Alqorithms 2019 US Report: New Challenaes to Government Use of 
Alaorithmic Decisión Systems, our second major report assessing recent court cases 
focused on government use of algorithms. 

We also host expert workshops and public events on a wide range of topics. Our annual public Ai 
Now Svmposium convenes leaders from academia, industry, government, and civil society to 
examine the biggest challenges we face as Al moves into our everyday lives. Recordings of the 
program are available online. 


More information is available at ainowinstitute.org 
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RECOMMENDATIONS 


1. Regulators should ban the use of affect recognition in important decisions that impact 
people's lives and access to opportunities. Until then, Al companies should stop 
deploying it. Given the contested scientific foundations of affect recognition 
technology-a subclass of facial recognition that claims to detect things such as 
personality, emotions, mental health, and other interior States—it should not be allowed to 
play a role in important decisions about human lives, such as who is interviewed or hired 
for a job, the price of insurance, patient pain assessments, or student performance in 
school. Building on last year's recommendation for stringent regulation, governments 
should specifically prohibit use of affect recognition in high-stakes decision-making 
processes. 

2. Government and business should halt all use of facial recognition in sensitive social 
and political contexts until the risks are fully studied and adequate regulations are in 
place. In 2019, there has been a rapid expansión of facial recognition in many domains. 
Yet there is mounting evidence that this technology causes serious harm, most often to 
people of color and the poor. There should be a moratorium on all uses of facial 
recognition in sensitive social and political domains—including surveillance, policing, 
education, and employment-where facial recognition poses risks and consequences that 
cannot be remedied retroactively. Lawmakers must supplement a moratorium with (1) 
transparency requirements that allow researchers, policymakers, and communities to 
assess and understand the best possible approach to restricting and regulating facial 
recognition; and (2) protections that provide the communities on whom such technologies 
are used with the power to make their own evaluations and rejections of its deployment. 

3. The Al industry needs to make significant structural changes to address systemic 
racism, misogyny, and lack of diversity. The Al industry is strikingly homogeneous, due in 
large part to its treatment of women, people of color, gender minorities, and other 
underrepresented groups. To begin addressing this problem, more information should be 
shared publicly about compensation levels, response rates to harassment and 
discrimination, and hiring practices. It also requires ending pay and opportunity inequality 
and providing real incentives for executives to create, promote, and protect inclusive 
workplaces. Finally, any measures taken should address the two-tiered workforce, in 
which many of the people of color at tech companies work as undercompensated and 
vulnerable temporary workers, vendors, or contractors. 

4. Al bias research should move beyond technical fixes to address the broader politics 
and consequences of Al’s use. Research on Al bias and fairness has begun to expand 
beyond technical Solutions that target statistical parity, but there needs to be a much more 
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rigorous examination of Al's politics and consequences, including cióse attention to Al’s 
classification practices and harms. This will require that the field center "non-technical" 
disciplines whose work traditionally examines such issues, including Science and 
technology studies, critical race studies, disability studies, and other disciplines keenly 
attuned to social context, including how difference is constructed, the work of 
classification, and its consequences. 

5. Governments should mándate public disclosure of the Al industry’s climate impact. 

Given the significant environmental impacts of Al development, as well as the 
concentration of power in the Al industry, it is important for governments to ensure that 
large-scale Al providers disclose the climate costs of Al development to the public. As with 
similar requirements for the automotive and airline industries, such disclosure helps 
provide the foundation for more informed collective choices around climate and 
technology. Disclosure should inelude notifications that allow developers and researchers 
to understand the specific climate cost of their use of Al infrastructure. Climate-impact 
reporting should be sepárate from any accounting for offsets or other mitigation 
strategies. In addition, governments should use that data to ensure that Al policies take 
into accountthe climate impacts of any proposed Al deployment. 

6. Workers should have the right to contest exploitative and invasive Al-and unions can 
help. The introduction of Al-enabled labor-management systems raises significant 
questions about worker rights and safety. The use of these systems—from Amazon 
warehouses to líber and InstaCart—pools power and control in the hands of employers 
and harms mainly low-wage workers (who are disproportionately people of color) by 
setting productivity targets linked to chronic injuries, psychological stress, and even death 
and by imposing unpredictable algorithmic wage cuts that undermine economic stability. 
Workers deserve the right to contest such determinations, and to collectively agree on 
workplace standards that are safe, fair, and predictable. Unions have traditionally been an 
important part of this process, which underscores the need for companies to allow their 
workers to organize without fear of retaliation. 

7. Tech workers should have the right to know what they are building and to contest 
unethical or harmful uses of their work. Over the last two years, organized tech workers 
and whistleblowers have emerged as a powerful forcé for Al accountability, exposing 
secretive contracts and plans for harmful producís, from autonomous weapons to 
tracking-and-surveillance infrastructure. Given the general-purpose nature of most Al 
technology, the engineers designing and developing a system are often unaware of how it 
will ultimately be used. An object-recognition model trained to enable aerial surveillance 
could just as easily be applied to disaster relief as it could to weapons targeting. Too 
often, decisions about how Al is used are left to sales departments and executives, hidden 
behind highly confidential contractual agreements that are inaccessible to workers and 
the public. Companies should ensure that workers are able to track where their work is 
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being applied, by whom, and to what end. Providing such information enables workers to 
make ethical choices and gives them power to collectively contest harmful applications. 

8. States should craft expanded biometric privacy laws that regúlate both public and 
prívate actors. Biometric data, from DNA to faceprints, is at the core of many harmful Al 
systems. Over a decade ago, Illinois adopted the Biometric Information Privacy Act (BIPA), 
which has now become one of the strongest and most effective privacy protections in the 
United States. BIPA allows individuáis to sue for almost any unauthorized collection and 
use of their biometric data by a prívate actor, including for surveillance, tracking, and 
profiling via facial recognition. BIPA also shuts down the gray and black markets that sell 
data and make it vulnerable to breaches and exploitation. States that adopt BIPA should 
expand it to inelude government use, which will mitígate many of biometric Al's harms, 
especially in parallel with other approaches, such as moratoriums and prohibitions. 

9. Lawmakers need to regúlate the integration of public and private surveillance 
infrastructures. This year, there was a surge in the integration of privately owned 
technological infrastructures with public systems, from "smart" cities to property tech to 
neighborhood surveillance systems such as Amazorfs Ring and Rekognition. Large tech 
companies like Amazon, Microsoft, and Google also pursued major military and 
surveillance contracts, further enmeshing those interests. Across Asia, Africa, and Latin 
America, múltiple governments continué to roll out biometric ID projeets that create the 
infrastructure for both State and commercial surveillance. Yet few regulatory regimes 
govern this intersection. We need strong transpareney, accountability, and oversight in 
these areas, such as recent efforts to mándate public disclosure and debate of 
public-private tech partnerships, contracts, and acquisitions. 1 

10. Algorithmic Impact Assessments must account for Al’s impact on climate, health, and 
geographical displacement. Algorithmic Impact Assessments (AlAs) 2 help governments, 
companies, and communities assess the social implications of Al, and determine whether 
and how to use Al systems. Those using AlAs should expand them so that in addition to 
considering issues of bias, discrimination, and due process, the isues of climate, health, 
and geographical displacement are included. 

11. Machine learning researchers should account for potential risks and harms and better 
document the origins of their models and data. Advances in understanding of bias, 
fairness, and justice in machine learning research make it olear that assessments of risks 
and harms are imperative. In addition, using new mechanisms for documenting data 
provenance and the specificities of individual machine learning models should also 
become standard research practice. Both Model Cards 3 and Datasheets 4 offer useful 
templates. As a community, machine learning researchers need to embrace these 
analyses and tools to create an infrastructure that better considers the implications of Al. 5 
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12. Lawmakers should require informed consent for use of any personal data in 

health-related Al. The application of Al in healthcare requires greater protections around 
data. Whilethe informed-consent process that biomedical researchers and healthcare 
professionals generally employ in clinical settings requires discussion of the risks and 
benefits involved, affirmative approval before proceeding, and reasonable opportunities to 
withdraw from the study ortreatment, engineers and scientists commonly create training 
sets by scraping contení from whatever public sources are available. In order to ensure a 
future that does not amplify and reinforce historie injustices and social harms, Al health 
systems need better informed-consent approaches and more research to understand 
their implications in light of systemic health inequities, the organizational practices of 
healthcare, and diverse cultural approaches to health. 
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EXECUTIVE SUMMARY 


ln last year's report, we focused on Al's accountability gap, and asked who is responsible when Al 
systems harm us, and how we might remedy those harms. Lack of accountability emerged as a 
real and substantial problem—one that governments, companies, and civil society were just 
beginning to grapple with, even as Al's deployment ¡nto sensitive social domains accelerated. 

This year we saw a wave of pushback, as community groups, researchers, policymakers, and 
workers demanded a halt to risky and dangerous Al. Al Now’s 2019 report spotlights these 
growing movements, examining the coalitions involved and the research, arguments, and tactics 
used. We also examine the specific harms these coalitions are resisting, from Al-enabled 
management of workers, to algorithmic determinations of benefits and social Services, to 
surveillance and tracking of immigrants and underrepresented communities. What becomes olear 
is that across diverse domains and contexts, Al is widening inequality, placing information and 
control in the hands of those who already have power and further disempowering those who 
dorít. The way in which Al is increasing existing power asymmetries forms the core of our 
analysis, and from this perspective we examine what researchers, advocates, and policymakers 
can do to meaningfully address this imbalance. 

In doing so, the following key themes emerge: 

The spread of algorithmic management technology in the workplace is increasing the power 
asymmetry between workers and employers. Al threatens not only to disproportionately 
displace lower-wage earners, but also to reduce wages, job security, and other protections for 
those who need it most. 

This year, we've seen the rapid acceleration of algorithmic systems that control everything from 
interviewing and onboarding, to worker productivity, to wage setting and scheduling. Whether 
inside Amazorfs fulfillment warehouses, behind the wheel of an Uber, or interviewing for their first 
job out of college, workers are increasingly managed through Al, with few protections or 
assurances. Al systems used for worker control and management are inevitably optimized to 
produce benefits for employers, often at great cost to workers. New research also suggests that 
lower-wage workers, especially workers of color, will face greater harm from labor automation in 
the years to come, whether from displacement or from the degradation of work, as workers are 
increasingly tasked with monitoring and tending to automated systems ratherthan completing 
the tasks themselves. 
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Community groups, workers, journalists, and researchers-not corporate Al ethics statements 
and policies—have been primarily responsible for pressuring tech companies and governments 
to set guardrails on the use of Al. 

Companies, governments, NGOs, and academic institutions continued to dedicate enormous 
efforts to generating Al ethics principies and statements this year. However, the vast majority of 
these say very little about implementation, accountability, or how such ethics would be measured 
and enforced in practice. In parallel, we have seen growing evidence demonstrating a sharp divide 
between ethics promises and practice. Increasingly, meaningful action toward Al accountability 
has come from workers, community advocates, and organizers. In many cases, these efforts are 
not Al-specific, but rather focus on long-standing discriminatory and harmful policies and how 
Al-enabled technologies amplify such harms. This ineludes examples such as criminal justice 
advocates working to halt the use of discriminatory predictive policing tools, tenants-rights 
groups opposing facial recognition in housing, and a coalition of Latinx activists, tech workers, 
and students exposing and protesting lucrative tech company contracts with military and border 
agencies. 

Efforts to regúlate Al Systems are underway, but they are being outpaced by government 
adoption of Al Systems to surveil and control. 

This year alone, we have seen numerous proposed laws and regulations targeting Al. In particular, 
facial recognition continúes as the focus of many debates. Last year, Al Now joined calis to 
severely limit the use of facial recognition. This year, organizers across the US led the charge, 
successfully campaigning to pass laws banning facial recognition in several cities. Presidential 
candidate Bernie Sanders even promised a nationwide ban, and members of the United States 
Congress proposed several bilis, including the Commercial Facial Recognition Privacy Act of 
2019, the Facial Recognition Technology Warrant Act, and the No Biometric Barriers Act of 2019. 
Outside the US, there is litigation over use of the technology by UK pólice, rulings under GDPR in 
the EU, and the Australian parliament ordering a complete pause on the use of a national face 
database. 

Yet despite growing public concern and regulatory action, the rollout of facial recognition and 
other risky Al technologies has barely slowed down. So-called "smart city" projeets around the 
world are consolidating power over civic life in the hands of for-profit technology companies, 
putting them in charge of managing critical resources and information. For example, Google's 
Sidewalk Labs project even promoted the creation of a Google-managed Citizen credit score as 
part of its plan for public-private partnerships like Sidewalk Toronto. And Amazon heavily 
marketed its Ring, an Al-enabled home-surveillance video camera. The company partnered with 
over 700 pólice departments, using pólice as salespeople to convince residents to buy the 
system. In exchange, law enforcement was granted easier access to Ring surveillance footage. 
Meanwhile, companies like Amazon, Microsoft, and Google are fighting to be first in line for 
massive government contracts to grow the use of Al for tracking and surveillance of refugees and 
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residents, along with the proliferation of biometric identity systems, contributing to the overall 
surveillance infrastructure run by prívate tech companies and made available to governments. 

Al systems are continuing to amplify race and gender disparities via techniques like affect 
recognition, which has no sound scientific basis. 

Recent research has highlighted the dire lack of diversity within the Al industry, as well as the vast 
demographic differences between the populations that benefit and profit from Al's efficiency and 
those that bear the cost of Al's biases and exploitation. The result: outcomes like the sexist Apple 
Card algorithms that triggered investigations by both the Senate Committee on Finance and the 
New York State Department of Financial Services. Rather than recognizing the scale and systemic 
nature of the problem, tech companies have responded to mounting evidence of bias and misuse 
by primarily focusing on narrow diversity Solutions. They have also attempted technical debiasing, 
working to "fix" algorithms and diversity data sets, even though these approaches have proven 
insufficient and raise serious privacy and consent concerns. Notably, neither approach addresses 
underlying structural inequalities. Ñor do the approaches address the growing power asymmetry 
between those who produce and profit from Al and those subjected to Al's applications. 

Affect recognition, a subset of facial recognition that claims to "read" our inner emotions by 
interpreting the micro-expressions on our face, has been a particular focus of growing concern in 
2019-not only because it can encode biases, but because it lacks any solid scientific foundation 
to ensure accurate or even valid results. This was confirmed in 2019 by the largest metastudy to 
date on the topic. 6 Critics also noted the similarities between the logic of affect recognition, in 
which personal worth and character are supposedly discernable from physical characteristics, 
and discredited race Science and physiognomy, which was used to claim that biological 
differences justified social inequality. Yet in spite of this, Al-enabled affect recognition continúes 
to be deployed at scale across environments from classrooms to job interviews, informing 
sensitive determinations about who is "productive" or who is a "good worker," often without 
people's knowledge. 

Growing investment in and development of Al has profound implications in areas ranging from 
climate change to the rights of healthcare patients to the future of geopolitics and inequities 
being reinforced in regions in the global South. 

Finally, our report highlights new concerns associated with the development of Al and its effect 
on areas ranging from climate change to healthcare to geopolitics. Throughout, we note the 
significant consequences of Al's use and development, and the danger of leaving determinations 
around these issues in the hands of a small number of individuáis and corporations, whose 
incentives and worldviews are often at odds with the ¡nterests of those who bear the 
consequences of such decisions. 
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The climate impact of Al development has become a particular area of concern, as recent 
research demonstrated that creating just one Al model for natural-language Processing can emit 
as much as 600,000 pounds of carbón dioxide. Key concerns have also emerged regarding how 
algorithmic health-management tools are impacting patient data and well-being, and the lives of 
those who care for them. The China "Al arms race” narrative also took center stage this year. In 
this report we examine the way it has been leveraged to paint efforts to regúlate and curb harmful 
Al as "anti-progress," even though such efforts focus on upholding the democratic valúes the US 
claims to promote in its relations with China. Finally, there is a need for more research on the 
locally specific real-world impact of the Al industry on countries in the global South and the ways 
it reinforces historical inequities in these regions. 

As with our previous reports, we present these findings and concerns in the spirit of engagement, 
and with the hope that we can contribute to a more holistic understanding of Al that centers the 
perspectives and needs of those most affected, and that shapes technical development and 
deployment to these ends. 
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1. THE GROWING PUSHBACK AGAINST HARMFUL Al 

1.1 Al, Power, and Control 

Through a combination of surveillance, predictive analytics, and integration ¡nto workplace 
systems such as interviewing, human resources, and supervisión, employers are implementing 
algorithmic systems to rank and assess workers, automatically set wages and performance 
targets, and even fire workers. 7 In almost every case, these systems are optimized from the 
perspective of business owners and rarely involve or inelude worker perspectives, needs, or 
considerations. Most algorithmic management tools, like most algorithmic decisión systems, lack 
meaningful opportunities for workers to understand how the systems work or to contest or 
change determinations about their livelihood. 

Worker Productivity, Al, and "The Rate" 

A growing number of employers rely on Al systems to manage workers and set workloads, 
accruing significant centralized power and control. For example, Amazon uses an Al system that 
sets performance targets for workers, a so-called "rate." 8 The "rate" is calculated automatically, 
and changes from day to day. If a worker falls behind, they are subject to disciplinary action. In 
many warehouses, termination is an automated process (not unlike being "kicked off" a 
gig-economy platform). 9 According to Abdi Muse, an organizer with Amazon warehouse workers 
in Minneapolis, if workers fall behind the algorithmically set productivity rate three times in one 
day, they are fired, however long they may have worked for the company, and irrespective of the 
personal circumstances that led to their "mistakes." 10 Muse recounts workers deciding between 
going to the bathroom and maintaining their rate. Many workers in the Amazon warehouse where 
he organizes are Somalí immigrants, whose work at Amazon helps send money home. This 
pressure is exacerbated by the low wages paid to many warehouse workers. A report from the 
Economic Roundtable found that in Californias Inland Empire, the home to a major Amazon 
warehouse hub, "86 percent [of Amazorís logistics employees] eam less than the basic living 
wage... The typical worker had total annual earnings in 2017 of $20,585, which is slightly over half 
of the living wage." 11 

Amazon isrít the only company using Al to enforce worker productivity. Chris Ramsaroop, a 
founding member of the organization Justicia for Migrant Workers, documents the integration of 
tracking and productivity technologies in the agriculture sector in Cañada, finding that 
"surveillance technologies are utilized to regiment workers to determine their pace at work and 
their production levels, much like what we see in warehouses." 12 When the Philadelphia Marriott 
Downtown began using an app to give its housekeepers room assignments, workers found the 
new system sent them zigzagging across a hotel the size of a city block. It reduced their ability to 
organize their day, making their work more physically demanding. 13 Reporting earlier this year also 
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revealed that the Children’s Hospital of Philadelphia (CHOP) hired outside contractors to 
assemble and distribute supplies like syringes, gauze, and other essential equipment and used an 
opaque algorithmic "rate" that set the amount of work. If anything is off, it's "nearly impossible to 
meet [the rate],” say workers, "if they're understaffed or overstaffed, if it’s a holiday, if there’s a 
person who's new and just getting up to speed.” And, if workers don’t meet their rate, they'll be 
written up. 14 

CHOP links the practice of hiring contract workers (whose labor is leased by one firm to another) 
to algorithmically set productivity rates. This mandates a rate of productivity as part of the 
contractual agreement and enforces that rate through an algorithm, instead of through on-site 
supervisors. 

Such rate-settlng systems rely on pervasive worker surveillance to measure how much they are 
doing. Systems to enable such invasive worker monitoring are becoming more common, 
including in traditionally "white-collar" working environments. For example, the start-up Humanyze 
incorporates sensors ¡nto employee badges to monitor employee activities, telling employers 
where workers go, whom they interact with, and how long they stay in a given place. Another 
company called Workplace Advisor uses heat sensors to achieve a similar aim. And though the 
usefulness of these producís is disputed, 15 they reflect an increasing willingness to engage in 
invasive surveillance of workers ¡n the ñame of workplace control and eking out incremental gains 
in productivity. 

Algorithmic Wage Control 

Algorithmic worker management and control systems have also had a severe negative ¡mpact on 
wages across the so-called "gig economy." These platforms treat workers as subjects of constant 
experimentation, often in ways that destabilize their economic and even psychological security. 16 
In many instances, industries that adopt discourses of technological advancement are driven by 
precarious worker labor-what Mary Gray and Siddarth Suri describe as ghost work. 17 

Such Al systems are correlated with low wages and "flexible" work policies that, in practice, often 
make it hard for workers to predict their income, schedule, or whether they will even be able to 
work that day. Similar to other algorithmic management systems, these function by pooling 
information and power together for the benefit of owners, managers, and a handful of developers, 
allowing companies to optimize such systems in ways that maximize revenue without regard to 
the need for stable and livable wages or predictable incomes, schedules, and availability of work. 
Indeed, many workers have reported being abruptly "kicked off a gig work platform, and finding 
themselves unable to work without warning. The process to reinstate an account can be obscure 
and onerous. 18 

These platforms are contlnually optimlzed by companies and owners. Abrupt changes intended 
to increase revenue for the company can result in significant losses for workers. In one example, 
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Instacart made changes to ¡ts interface that misled customers into thinking they were leaving a 
tip for workers, when in fact they were paying a Service fee to the company. 19 This practice is 
something that DoorDash also engaged in until July of this year. 20 

These examples demónstrate the significant power asymmetry between workers and customers 
on one hand, and the companies who control worker management platforms on the other. How, 
and where, companies may be "optimizing" their platforms at the expense of workers remains 
largely opaque to anyone outside of companies' corporate offices, and what is known comes 
largely from worker whistleblowers. 

The ability of automated management platforms to manipúlate (and arbitrarily cut) wages has 
been at the heart of worker grievances. Instacart workers report that their earnings decreased 
precipitously over the last year. 21 Uber and Lyft workers report similar drops. 22 Many identify this 
as part of a tactic to make workers dependent on the platform for wages, drawing them in with 
promises of a living wage and flexible working conditions, then severely cutting wages once 
workers have structured their lives around working for the platform. 

Legal scholar Veena Dubal, who has worked with Uber and taxi drivers, makes the case that these 
practices are not new, but "[reproduce] risky, early 20th century working conditions," enabled by 
large-scale Al platforms and deregulation. 23 As labor scholar Jim Stanford puts it, "The only thing 
truly new about gig employment is ¡ts use of digital and on-line techniques to assign work, 
discipline workers ... and control the money. That's more effective than the bulletin boards and 
classified ads of yesteryear—but it hardly negates the inherent power imbalance between an 
individual worker and the multibillion-dollar company they work for." 24 

It is important to note that such concerns do not always transíate outside of the US context, 
which has a history of social security and labor laws whose enforcement has provided some 
measure of worker protection. This is not the case in many global South regions, where the ability 
of residents to appeal to the State for resources, along with a history of poorly enforced social 
safety protections, mean that the role of Al-enabled platform work has a more complex set of 
implications, which cannot be read through a US-centric lens, and need to account for global 
histories of colonialism and ¡nequality 25 For instance, researchers Surie and Koduganti examine 
platform workers in Bengaluru, India, showing that they exist in a context in which many workers 
are already stitching together "flexible" work options, without the expectation of social safety nets. 
Given these existing practices and expectations, Al-enabled platform work provides a 
comparatively more lucrative source of employment. 26 Similarly, Indian Turk workers (who until 
recently made up 40 percent of the total platform workforce) benefited from the global pay rates 
set by the platform, and the cost of living difference between India and the US. 27 This meant that 
tasks that US workers would not take due to low pay were more attractive for their global South 
counterparts. In a study of beauty and wellness platforms in India, researchers discovered that 
women found these platforms attractive because they allowed them to pursue economic activity 
within the constraints of gendered, religious, and family norms. 28 In these ways, Al-enabled 
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platform work has challenged the boundedness of local labor markets globally, 29 and brought with 
it a diverse set of implications that cannot be understood as analogous to the US experience. 

Al in Hiring Tech 

Al systems to manage and control workers are also being applied in hiring, rapidly and actively 
shaping the labor market and helping determine who is fit for work, and who isrít. Most hiring 
tech operates in the absence of any specific rules or requirements to disclose their use for 
candidate selection, ranking, and hiring to the job seekers whose lives these Al systems affect. 30 
Commercial firms across industries, including major employers like Unilever, 31 Goldman Sachs, 32 
and Target, 33 are integrating predictive technologies into the process of selecting whom they hire. 
Al systems also actively shape employment advertising, résumé ranking, and assessment of both 
active and passive recruitment. 34 

Because Al systems often encode and reproduce patterns of bias within categories such as 
"competence,” "success," and "cultural fit," the rapid deployment of such systems in hiring has 
significantly raised the stakes of their use. 35 Indeed, many researchers suspect that these tools 
most likely exacérbate inequity and reinforce discrimination, creating what legal scholar Pauline 
Kim terms "classification bias." 36 But without meaningful access to these systems and their 
processes, workers lack the evidence necessary to challenge their use. 37 

Often an early mover on these issues, the State of Illinois has already passed a law pushing back 
on the secrecy of these systems. So far, it is the only State to do so. Scheduled to take effect in 
January 2020, the Artificial Intelligence Video Interview Act mandates that employers notify job 
candidates when artificial intelligence is used in video interviewing, provide an explanation of how 
the Al system works and what characteristics it uses to evalúate an applicant's fitness for the 
position, obtain the applicant's consent to be evaluated by Al before the video interview starts, 
limit access to the videos, and destroy all copies of the video within 30 days of an applicant's 
request. 38 

Faced with pushback, hiring tech vendors are also attempting to make the case that their 
systems help fight against historical and human biases, claiming they have been designed to 
reduce discrimination and increase diversity. Yet at this point, such claims amount to marketing 
statements and are unsupported by peer-reviewed research. Instead, studies show that therejust 
isrít enough transparency to assess whether and how these models actually work (and to what 
effect)-let alone to determine whether they’re unbiased. 39 

This led the Electronic Privacy Information Center to file a complaint with the Federal Trade 
Commission alleging that one Al hiring company, PlireVue, is engaging in "unfair and deceptive" 
business practices by failing to ensure the accuracy, reliability, or validity of its algorithmically 
driven results. 40 
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Employers, not workers, are the "customers" whom Al hiring companies seek to court with 
promises of efficiency and fewer worries about accountability and liability. In fact, several 
prominent companies, like pymetrics, actively offerto cover any of their customers' legal fees or 
liabllltles that might arise from the use of their producís or Services. 41 

Al-driven hiring systems are only the starting point of a concentrated push to use Al to monitor 
and control workers and workplaces—as Ifeoma Ajunwa and Daniel Greene put it, these platforms 
"create a managerial framework for workers as fungible human capital, available on demand and 
easily ported between job tasks and organizations." 42 

It's critical that researchers and advocates not only examine the application of artificial 
intelligence in the hiring process in isolation, but also consider how Al is being implicated in 
broader shifts in labor practices, and how it might be serving to define and redefine notions of 
competence and ability 43 

Labor Automation's Disparate Impacts 

In recent years, two predominant narratives have emerged around the future of work and labor 
automation. 44 One insists that labor automation will yield a net gain for society—increasing 
productivity, growing the economy, and creating more jobs and demand for workers that will 
offset any technological displacement that happens along the way 45 The other predicts a labor 
apocalypse, where robots will ultimately take over the workforce, create massive unemployment, 
and serve only the financial interests of those who own them and the engines of our economy 46 
Both narratives are predicated on the assumption that automation in the workplace is inevitable 
and that automated systems are capable of performing tasks that had previously been the work 
of humans. 

What is missing from both conflicting narratives isthe more nuanced prediction of who will be 
harmed and who will benefit from labor automation in the years to come. This year, more data 
emerged that begins to provide the following answer to those questions: labor automation and 
the corresponding restructuring and reduction in waged work will likely disproportionately harm 
Black, Latinx, and low-wage workers in the US. 

One such study from the Brookings Institute predicts that certain demographic groups will likely 
bear more of the burden of adjusting to labor automation than others, implying that the benefits 
of automation-increased efficiency and profit-are not shared with all workers, but acometo 
those atthetop. 47 

First, the study found that lower-wage workers stand to lose the most due to automation while 
white-collar workers will likely remain largely unaffected. Using a model that views a job as a 
bundle of tasks (some of which can be automated and others not), Brookings concluded that the 
average "automation potential" for US occupations requiring less than a bachelor's degree is 55 
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percent-more than double the 24 percent susceptibility among occupations requiring a 
bachelor's degree or more. 48 That means US workers in occupations that pay the least, like food 
preparation and serving, production jobs in faetones, and administrative support-which pay 
wages of only 50 to 75 percent of the national average-could experience 60 to 80 percent 
task-level disruption. 49 Meanwhile, higher-paying jobs in business and financia! operations or 
engineering, where US workers earn 150 percent of the average wage, will likely experience as 
little as 14 percent of their current tasks being displaced by automation. 

This has serious implications in terms of the risk exposure faced by certain communities. Black, 
Native American, and Latinx workers who make up a larger proportion of the workforce in 
occupations like construction, agriculture, and transportaron 50 face average task-automation 
potentials of 44 to 47 percent. That's anywhere from five to eight percent more than their White 
counterparts. 51 

The disparate effeets of task automation will also likely entail disproportionate job losses. Even 
McKinsey & Company, which believes Al could lift productivity and economic growth, concluded 
that labor automation will further exacérbate the racial wealth gap in the US absent any 
interventions. 52 One study from July 2019 found that more than a quarter of Latinx workers—as 
many as seven million people-are in jobs that could be automated by 2030. 53 That translates to a 
potential displacement rate of 25.5 percent for Latinx workers, three percentage points higher 
than the national average. 54 McKinsey calculated that 4.6 million Black workers will be displaced 
by 2030 due to automation, with a potential displacement rate of 23.1 percent. 55 

The exact job-loss figures caused by automation are ultimately hotly contested. After MIT 
Technology Review synthesized 18 different reports on the effeets of automation on labor with 
predictions ranging from a gain of nearly 1 billion jobs globally by 2030 to a loss of 2 billion, it 
aptly noted that "prognostications are all over the map." 56 With all of these projections, the devil is 
in the details. We may "have no idea how many jobs will actually be lost to the march of 
technological progress,” 57 but we can begin to answer who will lose their jobs based on the power 
dynamics and economic disparities that already exist today. 

The Limits of Corporate Al Ethics 

This year, many companies, governments, NGOs, and academic institutions followed the familiar 
path of generating Al ethics principies and statements. These primarily Western entities—many of 
them driven by an industry that is predominantly White, male, and wealthy-often present such 
ethics principies as the product of a growing "global consensus” on Al ethics. This promotes a 
majoritarian view of ethics, which is especially concerning given the widespread evidence 
showing that Al bias and misuse harms the very people whose voices are largely missing in 
ethics debates. 58 
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There are now so many ethics policy statements that some groups began to aggregate them ¡nto 
standalone Al ethics surveys, which attempted to summarize and consolídate a representativa 
sample of Al principie statements in orderto identify themes and make normative assertions 
about the State of Al ethics. 59 Despite the ¡ncrease in Al ethics contení and corresponding 
meta-analyses, ethical principies and statements rarely focus on how Al ethics can be 
implemented and whether they're effective. Similarly, such Al ethics statements largely ignore 
questions of how, where, and by whom such guidelines may be operationalized. 

These surveys tend to aggregate Al ethics contení from a very wide variety of contexts, blending 
corporate statements released on corporate blogs, 60 publicly informed governing declarations, 61 
government policy guidelines from national and coalition strategies, 62 and nonprofit mission 
statements and charters. 63 However, they usually lack a comprehensive account of the methods 
used and sometimes equate internal and often secret corporate decision-making processes with 
grassroots-driven statements and governmental policy recommendations. 

The vast majority of these documents were generated from countries and organizations in the 
global North. 64 Principie statements and the ethical priorities of the global South with regard to 
artificial intelligence are often absent from these surveys. 

Scholars and advocates have increasingly called attention to the gap between high-level 
statements and meaningful accountability. 65 Critics have identified conflicting ideáis and vague 
definitions as barriers that are preventing the operationalization of ethics principies in Al product 
development, deployment, and auditing frameworks. In "Principies Alone Cannot Guarantee 
Ethical Al," philosopher and ethicist Brent Mittelstadt makes the observation that, unlike medicine, 
Al has no formal professional governance structure or norms-no agreed-upon definitions and 
goals forthe field. 66 Most importantly, unlike medical ethics, Al ethics has no external oversight or 
standard protocols for enforcing ethical guardrails. 

This lack of professional and legal accountability undermines corporate ethics approaches. One 
example is Microsoft's funding of an Israeli facial-recognition surveillance company called 
AnyVision that targets Palestinians in the West Bank. 67 AnyVision facilitates surveillance, allowing 
Israeli authorities to identify Palenstenian individuáis and track their movements in public space. 
Given the documented human-rights abuses happening on the West Bank, 68 together with the 
civil-liberties implications associated with facial recognition in policing contexts, 69 at a mínimum, 
this use case directly contradicts Microsoft's declared principies of "lawful surveillance" and 
"non-discrimination," along with the company's promise not to "deploy facial recognition 
technology in scenarios that we believe will put freedoms at risk.” 70 More perplexing still is that 
AnyVision confirmed to reporters that their technology had been vetted against Microsoft's ethical 
commitments. After public outcry, Microsoft acknowledged that there could be a problem, and 
hired former Attorney General Eric Holder to investígate the alignment between AnyVisiorís 
actions and Microsoft's ethical principies. 71 
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ln another of many such examples of corporations openly defying their own ethics principies, and 
despite declaring as one of its Al principies to "avoid creating or reinforcing unfair bias," 72 Google 
set up the Advanced Technology External Advisory Council (ATEAC), an ethics board that included 
Kay Coles James, the president of the Heritage Foundation and someone known for her 
transphobic and anti-immigrant views. Workers and the public objected. A petition signed by over 
2,500 Google workers argued: "ln selecting James, Google is making clearthat its versión of 
'ethics' valúes proximity to power over the wellbeing of trans people, other LGBTQ people, and 

immigrants_Not only are James' views counterto Google's stated valúes, but they are directly 

counter to the project of ensuring that the development and application of Al prioritizes justice 
over profit." 73 Following the backlash, Google dissolved ATEAC after a little over a week. 74 

Yet even if one believes that corporate Al ethics might help guide better tech practices on some 
level, it is clearthat change in the design, development, and implementation of Al systems largely 
occurs when there is pressure on companies from workers, the press, and policymakers. For 
example, the various controversies Facebook has publicly faced demónstrate that public pressure 
and organized workers appear to be far better at ensuring ethical Al than principies. Facebook 
advertises its own internal ethics process. However, investigative reports from outlets such as 
ProPublica on Facebook’s discriminatory online advertising filtering mechanisms together with 
published studies about Facebook's online ad ecosystem bolstered lawsuits brought by the 
Department of Urban Housing and Defense, civil rights groups, and labor organizations against 
the company in 2019.® 

Given the concerns that ethical promises are inadequate in the face of urgent accountability gaps, 
many have argued that human rights principies, which are based on more established legal 
interpretations and practice, should replace "ethics” as the dominant framework for conversations 
about Al governance and oversight. Advocates for this approach describe human rights as ethics 
"with teeth," or an alternative to the challenge of operationalizing ethics. 76 

The human rights legal framework has its own potential shortcomings, especially as it relates to 
Al technology. 77 One of these limitations is the challenges of enforcement of international human 
rights law when it pertains to powerful nations. Given that the US and China are considered global 
Al leaders that have both engaged in varying degrees of documented human rights abuses 
without facing meaningful consequences under international human rights law, 78 expecting 
human rights frameworks to constrain governmental and corporate actors within the countries 
currently dominating Al development may be impractical. Indeed, human rights law is mainly 
focused on government actors, so beyond the current lack of enforcement, the question of how it 
might serve to curb corporate malfeasance remains unanswered. 

By claiming a commitment to ethics, companies implicitly claim the right to decide what it means 
to "responsibly” deploy these technologies, and thus the right to decide what "ethical Al" means 
for the rest of the world. 79 As technology companies rarely suffer meaningful consequences when 
their ethical principies are violated, true accountability will depend on workers, journalists, 
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researchers, policymakers, and the public continuing to be at the forefront of the fight against the 
harmful uses of this technology. 

Some advocates are also pushing to ensure that engineers and developers are trained in ethics, 
and thus, the thinking goes, better capable of making more ethical decisions that can ensure 
more ethical tech. Barbara Grosz, a professor of natural Sciences, ¡magines a world in which 
"every time a Computer scientist logs on to write an algorithm or build a system, a message will 
flash across the screen that asks, 'Have you thought about the ethical implications of what you're 
doing?'" 80 The Design Justice Network takes this further, centering justice, not ethics, and calling 
on developers and designers to center affected communities in the process of creating 
technology together. 81 

Al developers and researchers make important determinations that can affect billions of people, 
and helping them consider whom the technology benefits and harms is important. The case of 
Uber's self-driving car makes clear what could have been had engineers, designers, and 
executives put more care into ethics and safety (although whether or not these were decisions 
engineers had the power to make is not something we know). In 2018, an autonomous Uber in 
Arizona killed Elaine Herzberg, a pedestrian. A recent National Transportation Safety Board 
investigation found significant problems with Uber's autonomous system, including a shocking 
disclosure that Uber's self-driving software "wasrft designed to expect that pedestrians outside 
crosswalks may be Crossing the Street." 82 Similar engineering failures led to over 37 accidents 
involving autonomous Uber vehicles. 83 It is clear that better testing and engineering practices, 
grounded in concern forthe implications of Al, are urgently needed. 

However, focusing on engineers without accounting for the broader political economy within 
which Al is produced and deployed runs the risk of placing responsibility on individual actors 
within a much larger system, erasing very real power asymmetries. Those at the top of corporate 
hierarchies have much more power to set direction and shape ethical decision-making than do 
individual researchers and developers. Such an emphasis on "ethical education" recalls the push 
for "unconscious bias" training as a way to "improve diversity." Racism and misogyny are treated 
as "invisible" symptoms latent in individuáis, not as structural problems that manifest in material 
inequities. These formulations ignore the fact that engineers are often not at the center of the 
decisions that lead to harm, and may not even know about them. For example, some engineers 
working on Google's Project Maven weren't aware that they were building a military drone 
surveillance system. 84 Indeed, such obscurity is often by design, with sensitive projects being split 
into sections, making it impossible for any one developer or team to understand the ultímate 
shape of what they are building, and where it might be applied. 85 
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How Al Companies Are Inciting Geographic Displacement 

Just as the development environments of artificial ¡ntelligence and machine learning are filled 
with disparities, so too are the broader cityscapes in which their development takes place. 
Whether within large suburban tech campuses or smaller urban tech start-ups, Al and machine 
learning environments are never contained within company walls. Rather, the racial, gendered, 
and class-based biases well proven to exist within Al labs are porous, spilling into external spaces. 
Often this results in processes popularly described as tech-driven gentrification, or the 
replacement of poor, working-class, and/or racialized residents with wealthier and whitertech 
employees. 

While numerous cities have experienced Al displacement, San Francisco has been especially 
impacted. With the IPO releases of a number of tech companies this past year, the real estáte 
industry has predicted a new surge of tech wealth. As during the dot-com boom, speculators 
disproportionately evict Black and Latinx working-class tenants in order to create new housing for 
wealthier and whitertech employees. 86 

Meanwhile, across the Bay, Alameda and Contra Costa counties, which were devastated by the 
2008 foreclosure crisis, continué to see the loss of Black and Latinx homeownership and housing. 
In fact, the subprime crisis and the fintech derivatives market it relied upon can also be 
understood as a technology of Al displacement. 87 The very algorithms used by lenders and banks 
relied upon codifying Black homeowners as explotable. 88 In the post-2008 era, Wall Street 
investment firms such as Blackstone/lnvitation Homes use machine learning systems to 
calcúlate rental acquisitions, buying up huge swaths of property foreclosed during the subprime 
crisis and renting them out as single-family homes. 89 They rent such homes out today using 
proptech Al management systems and property databases known to engage in tenant profiling 
that disfavors people of color. 90 

This era has also been marked by the 2008 launch of Airbnb, the San Francisco start-up linked to 
ongoing gentrification of cities worldwide as long-term tenants are replaced with tourists. 91 Even 
single room occupancy hotels (SROs), which have historically housed precarious residents, have 
been converted into "tech dorms" and tourist accommodations in cities such as San Francisco 
and Oakland. 92 

Also characteristic are the prívate tech luxury buses that facilítate reverse commuting of tech 
workers to Silicon Valley from urban centers. Landlords have found property adjacent to "Google 
bus stops" lucrative, leading to increased rental prices and evictions along with new luxury and 
market-rate development projects. 93 94 

While eviction trends often appear bleak, gentrification has not transpired without resistance. As 
during the dot-com boom and foreclosure crisis, numerous organizations and collectives formed 
to organize for housing justice. Existing housing organizations also rose to the occasion, and new 
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efforts, such as the Anti-Displacement Coalition in San Francisco and the Bay Area-wide Tenant 
Organizing NetWork, were formed. Rent-control protection groups and tenant unions have been 
forming monthly, and statewide tenant organizing has been on the rise. 95 Al displacement is not 
limited to San Francisco, or even to the United States, where most major Al companies reside. For 
instance, Cluj, Romanía, the supposed "Silicon Valley of Eastern Europe,” has become one of many 
Al and IT outsourcing locales. 96 The laborers there often make more than their neighbors. 
Increasingly, they are able to rent and buy fancier fíats adjacent to outsourcing offices, which has 
incentivized the City to evict racialized Roma residents. Accordingly, housing justice groups such 
as Social Flousing Now (Casi Sociala Acum) are in the midst of organizing against evictions and 
for the development of social housing. 97 

Back in the North, there have been new forms of international solidarity in the works against Al 
displacement. For instance, current organizing against Google's proposed new campus in San 
José is being led by groups such as Serve the People San José. As they have argued, Google's 
new campus will lead to mass displacement and unaffordability. Thus they have been organizing 
marches, Google bus blockades, and City Council demonstrations. 98 Much of this has taken place 
in solidarity with organizers and groups in Berlín such as Google Is Not a Good Neighbor (Google 
ist kein guter Nachbar), which in 2018 collectively blocked Google from launching a new tech 
campus in the neighborhood of Kreuzberg. 99 Solidarity has also been found among New York City 
organizers who successfully fought the development of a new Amazon campus in 2019, and with 
activists in Toronto committed to thwarting gentrification induced by Sidewalk Labs. 100 

During demonstrations, banners, light projections, video clips, and statements of support have 
expressed international solidarity, revealing a newtrend toward urban justice. 101 Much work 
remains to link struggles against forms of tech-sector displacement worldwide. 


7.2 Organizing Against and Resisting Consolidations of 
Power 

Organizing and Pushback 

Pushback against Al isrít new, ñor is it confined to tech companies and elite universities. Often, it's 
not even identified as related to biased and harmful Al. This is in part because Al systems are 
often integrated "in the backend,” as part of operationalizing larger policies which themselves are 
the focus of organizing. Because Al technologies are often applied in ways that amplify and 
exacérbate historical patterns of inequality and discrimination, it is these historical practices—not 
Al systems alone-to which organizers and communities seeking justice are reacting. 
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Community Organizing 

Community organizers have been an ¡mportant forcé in the pushback against harmful Al. This 
was most visible in the wave of community organizing this year tackling the use of facial 
recognition in cities around the world: San Francisco, 102 Oakland, 103 Somerville, 104 Montreal, 105 and 
Detroit, 106 among others. Community-driven organizing led directly to bans on facial recognition in 
many of these localities. As we highlight elsewhere in this report, in Brooklyn, tenants of Atlantic 
Plaza Towers organized and successfully challenged the incorporation of a facial recognition 
system into their building. 107 

Community organizers also played a critical role in mapping the connections between mass 
incarceration, the surveillance of communities of color, and the push to adopt predictive policing 
tools. In Los Angeles, community organizers successfully advocated for a temporary suspensión 
of the Los Angeles Pólice Department's use of the predictive policing program LASER, which 
purported to identify individuáis likely to predict violent crimes. The Stop LAPD Spying Coalition 108 
argued that the department used proxy data to discrimínate against Latinx and Black community 
members. 109 In this effort, they were joined by UCLA students who signed a public-facing letter 
denouncing UCLA research and development of the predictive policing tool PredPol. Citing 
evidence of the role of such tools in perpetuating the overpolicing of communities of color, they 
requested UCLA researchers abstain from further development and commercialization of the 
tool. 110 Here we see students and communities acting to operationalize a critique of predictive 
policing that some Al researchers have made in recent years. 111 We discuss the growing student 
movement against harmful tech and exclusionary tech cultures in more detail below. 

In St. Louis, Missouri, residents also demonstrated against policing tech, protesting a proposed 
agreement between St. Louis pólice and a company called Predictive Surveillance Systems to 
deploy surveillance planes to collect images of citizens on the ground. They asserted that the 
"suspicionless tracking" would be an invasión of citizens' privacy. 112 

In Kansas, New York, Pennsylvania, and Connecticut, parents opposed the use of a web-based 
educational platform called Summit Learning in their schools. High schoolers staged sit-ins, and 
parents protested at school board meetings, emphasizing that the work of teachers could not be 
outsourced to technology-based platforms. And in Pennsylvania and Connecticut, they were 
successful in getting the Summit programs cut. 113 

The community group Mijente, which describes itself as a political home for multiracial Latinx and 
Chicanx people, has been at the forefront of mapping the connections between Al and 
immigration, and building broad coalitions. In July, Mijente joined Media Justice (an organization 
at the helm of San Francisco's facial-recognition ban) 114 and Tech Workers Coalition 115 to host 
Take Back Tech. The event convened community organizers alongside tech workers and 
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students, aiming to share strategies and knowledge, and to build coalitions between those 
harmed by oppressive technologies and those cióse to the research and development of such 
tech. 116 

In August of this year, Mijente released a detailed report based on FOIA record requests 
illuminating the central role certain technologies have played in detaining Black and Brown people, 
and the use of these technologies in immigration enforcement. 117 The organizaron also 
spearheaded the #NoTechforlCE campaign in opposition to the Trump Administratiorís raids and 
mass deportations of migrants along the Southern border of the US. This work helped shed light 
on lucrative tech company contracts with military and border agencies, and mobilized tech 
workers and students, while also emphasizing the human cost of a deportation campaign rife 
with human rights abuses. 118 Protesters catalyzed by the campaign have held regular 
demonstrations at Palantir's headquarters in Palo Alto and at its New York City offices. 119 

Organizations such as Never Again Action, 120 and Jews for Racial and Economic Justice 
(JFREJ) 121 have also led highly visible actions against Amazon, organizing Street protests and 
sit-ins in Amazon bookstores to protest against the company's ongoing work providing cloud 
computing Services to ICE. 122 And Immigrant rights groups such as Make the Road New York, 123 
along with Mijente, JFREJ, and other advocates, have reached out to academics and Computer 
Science and technology professionals through petitions, demanding that prominent conferences 
drop Palantir as a sponsor, given the company's role in empowering ICE. 124 Community-organized 
opposition to Palantir's role in ICE's detention of immigrants resulted in UC Berkeley's Privacy Law 
Scholars Conference, 125 Lesbians Who Tech, 126 and the Grace Plopper Celebration all pulling 
Palantir as a sponsor. 127 

Athena, a recently launched coalition, takes this further. Targeting Amazon, the coalition ineludes 
groups like ALIGN, New York Communities for Change, Make The Road New York, Desis Rising Up 
and Moving, and many others who successfully campaigned to challenge Amazorís plans to build 
its HQ2 in Queens, New York. 128 The campaign against Amazon's HQ2 was notable for its broad 
multi-issue approach, and for its somewhat unexpected success. Advocates and community 
organizers criticized the company's tax avoidance, the displacement that would follow in the wake 
of such a massive tech company headquarters, and the lavish corporate subsidies that New York 
offered the company. But they also organized around issues like Amazorís treatment of 
warehouse workers and its sale of surveillance tech. 129 Athena expands on this multi-issue 
approach, recognizing that Amazon is at the heart of a set of interlocking issues, including worker 
rights at warehouses, climate justice, and mass surveillance. The coalition ineludes organizations 
with experience across these domains, and is working to unify the growing opposition to the 
company and develop strategies capable of tackling Al companies whose reach extends into so 
many sensitive domains. 130 

These are only a handful of instances where community organizers are pushing back against Al 
and oppressive tech. Collectively, they highlight that the pushback against Al is not necessarily 
just about Al, but about policies and practices that exacérbate inequality and cause harm to our 
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communities. They also demónstrate that Al does not exist in isolation-it builds upon historical 
surveillance and policing practices that predominantly impact Black communities, communities 
of color, and the poor. 131 

Acknowledging and making these processes visible is an important step toward decentering 
technology in this conversaron. A focus on Al systems can obscure and abstract what are 
fundamentally institutional decisions masked by a technical veneer that exeludes the 
communities most affected from having a voice in the process. The pushback against Al thus 
builds upon the social justice work that organizers have engaged in for a much longer time. 
Researchers can play an important role in this conversaron by demystifying Al systems, pushing 
back on discourses that privilege technology, and listening closely to the communities leading 
these efforts. 132 

Worker Organizing 

Although organizing among tech workers has been underway for many years (spurred initially by 
contract workers), worker organizing around the harms of Al is relatively new. Such organizing is 
situated within a broader effort to address overall worker issues, ranging from wages and working 
conditions to concerns about respect, diversity, and inclusión, that seekto directly confront hostile 
workplace cultures. This broad organizing platform has resulted not only from coordinated efforts 
between cross-sector worker groups but also from the increased realizaron that workers and 
communities generally considered sepárate or distinct share common concerns when it comes to 
Al and large-scale technical systems. 

For example, tech workers have joined with community organizers in pushing back against tech's 
role in perpetuating human rights abuses and maltreatment of migrants and Latinx residents at 
the Southern US border. Since the fall of 2018, workers at Sal esforcé, 133 Microsoft, 134 Accenture, 135 
Google, 136 Tableau, 137 and GitHub 138 all signed petitions and open letters protesting their 
companies’ contracts with ICE. Developer Seth Vago pulled his open-source code out of the 
codebase used by the company Chef after learning of the company’s contract with ICE. 139 This led 
Chef to commit to cancel their contract, and spurred a larger discussion about the ethical 
responsibility of developers. 

Even workers at Palantir, the tech company at the center of ICE’s detention and tracking 
operations, circulated two open letters, and have expressed mistrust of and frustration with the 
company's leadership for its decisión to keep its contract with ICE. 140 Palantir CEO Alex Karp has 
publicly defended this work, 141 and in August the company renewed a contract worth $49 million 
overthree years. 142 

Other workers protested the development of military Al systems: Microsoft employees signed an 
open letter to the company asking it not to bid on JEDI, a major Department of Defense 
cloud-computing contract, which the company ultimately won. 143 In February, employees at the 
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company followed this with a cali to cancel a $480 milllon contract to provide augmented reality 
headsets to the US military, saying they did "not want to become war profiteers." 144 

Such examples demónstrate how tech workers' strategies build solidarity with the communities 
most affected by Al's harmful uses. Worker organizing around Al is also part of a broader 
tech-worker movement focused on a broad range of social justice issues, including 
displacement, 145 two-tiered workforces and the exploitation of contract workers, 146 and climate 
change. In April, 8,695 Amazon workers publicly signed a letter calling on the company to address 
its contributions to climate change through a shareholder resolution, 147 and staged a walkout in 
September in the face of inaction by the company. 148 The September climate walkout was the first 
labor action coordinated across múltiple tech companies, and provides an indication of the 
growth of tech-worker organizing during 2019. 

While the Google Walkout was, so far, the largest global labor action in tech, 149 it was only the first 
of many. Employees at Riot Games walked out in protest of the company's stance on torced 
arbitration, following allegations by múltiple employees that the company violated California's 
Equal Pay Act and claims of gender-based discrimination and harassment. 150 In China, developers 
protested what they described as the 996 schedule—9 a.m. to 9 p.m., six days a week—through a 
GitHub repository of companies and projects asking for excessive hours. 151 And in November, 
Google workers again walked out, hosting a rally of hundreds of workers in San Francisco 
protesting retaliaron against two organizers. 152 Following this rally, Google fired four organizers, 
signaling both the growing power of such efforts to irmpact Google and the company's intolerance 
of them. 153 

Another key development was contract workers leading the recent wave of tech-worker 
organizing, 154 centering the risks they experience as a result of the two-tier labor systems in which 
they work. In particular, contract workers lackthe benefits, stability, and pay of their employee 
colleagues, a disparity often enacted along racial lines. A 2016 report from Working Partnerships 
USA found that 58 percent of blue-collar contract workers in tech are Black and Latinx, and make 
an average of $19,900 annually. The report found that only 10 percent of "employee" tech workers 
are Black or Latinx, and that these workers make over $100 thousand annually. 155 Tech workers 
have called for an end to such discrimination, noting the racial divide and its implications for the 
perpetuation of structural inequality. 156 

In spite of the precarity and disadvantages that come with being classified as a contract worker, 
these workers continued to organize, from temp workers at Foxconn factories protesting unpaid 
wages and bonuses promised to them by recruitment agencies 157 to workers at Amazon 
warehouses walking out on Prime Day and successfully winning compromises to improve 
conditions. 158 Beyond protesting workplace conditions, contract workers have been leaders in 
pushing for ethical company practices, with Amazon-owned Whole Foods workers publishing a 
letter demanding Amazon end its involvement with ICE 159 and sharing a video revealing the 
company's union-busting tactics. 160 
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Such organizing has led to a wave of unions forming among workers on the corporate campuses 
of tech firms in recent years, a trend that started in 2014, well before white-collar workers began 
visibly organizing. These included food-service workers at Airbnb, 161 Facebook, 162 and Yahoo, 163 
and shuttle drivers and security guards at a host of Silicon Valley firms. 164 In Poland, Spain, and 
Germany, unionized Amazon warehouse workers held strikes to demand higher pay and better 
working conditions. 165 

But Amazon and other tech companies are using tactics to prevent unions from forming: for 
example, 14 software engineers at the start-up Lanetix were fired shortly after unionizing earlier 
this year. The workers filed charges with the National Labor Relations Board and ultimately won 
their case. 166 Google also hired a Consulting firm known for its anti-union work amid employee 
unrest, a fact disclosed by whistleblowers. 167 

Globally, strikes by transport workers grew in response to ride-sharing apps that are decreasing 
wages and living standards. líber drivers staged major strikes in cities around the globe 168169170 
including drivers' occupying Uber’s offices in France, 171 while Ola drivers in India protested 
decreasing driver incentives amid increasing fuel prices. 172 China Labor Bulletin recorded nearly 
1,400 transport worker protests over a five-year period in cities across the country. 173 

In the State of California, driver protests resulted in significant and tangible gains-though not 
from the companies themselves. Instead, Californias State Assembly passed Assembly Bill 5 
(AB5), which makes it much harder for companies like Uber to label workers as independent 
contractors, granting them basic worker protections. 174 In arguing against the change, Uber 
claimed that drivers weren’t core to Uber's business, and thus the company should not have to 
reclassify them as employees. 175 Based on this argument, Uber and Lyft appear likely to take their 
case to court. 176177 AB-5 in California was followed swiftly by a ruling in New Jersey that argued 
Uber had misclassified drivers as independent contractors, and demanded the company pay $649 
million in unpaid employment taxes. 178 

At the cióse of 2019, the pushback against the two-tier labor system appears poised to expand 
more widely: responding to worker protests, a group of US senators wrote Google CEO Sundar 
Pichai expressing objection to the company's heavy reliance on temporary workers (over half its 
workforce) 179 and urging the company to end its abuse of worker classifications. 180 Such 
reclassification of workers would result in thousands of people gaining access to essential 
benefits, workplace protections, and stability, which are denied contract workers. A move to 
reclassify all workers as employees would also have significant implications for the production 
and maintenance of Al systems, since low-paid contract workers are an essential labor forcé 
labeling Al training data, and moderating contení on large algorithmically driven platforms. 
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Student Organizing 

ln organizing against contracts with ICE and military Al, community organizers and workers were 
joined by students. Engineering students in particular have significant leverage, given that tech 
companies compete to recruit top talent and view them as "future workers." 181 Ethically minded 
students are having an impact on recruiting. Facebookalready has seen its offer acceptance rate 
dwindle from 85 percent to 35-55 percent at top Computer Science schools, as students begin to 
look beyond compensation and reflect on the commitment to ethics, diversity, and accountability 
demonstrated by the companies they hope to join. 182 

In the fall of 2018, students at Stanford first circulated a pledge not to accept interviews from 
Google until the company canceled its work on Project Maven, a US military effort to build 
Al-enabled drone surveillance, and committed to no further military involvement. 183 This 
movement grew significantly during 2019, spearheaded by Mijentés #NoTechForlCE campaign. 
Students around the US demonstrated against recruiting events on campus by technology 
companies known to be supporting border control or policing activities, such as Amazon, 
Salesforce, and Palantir. 184 Over 1,200 students representing 17 campuses signed a pledge 
asserting they would not work at Palantir because of its ties to ICE. 185 

In February, students from Central Michigan University fought against the creation of a university 
Army Al Task Forcé that was poised to endorse the military use of Al. 186 

Today's growing student movement targeting tech and military recruitment recalls historie 
student demonstrations against recruiting efforts, such as those by the CIA. In the 1960s, Brown 
University and Stanford University students disrupted CIA on-campus interviews in opposition to 
their involvement in Vietnam 187 . Similarly, University of Connecticut students in the early 1970s 
refused to accept the inclusión of Dow Chemical and Olin Mathieson in their on-campus events, 
as the Chemical engineering companies contributed to Chemical defoliants used in the Vietnam 
War. This movement later grew into an all-out campaign demanding the university's divestment 
from the military-industrial complex, and affiliated organizations such as the on-campus Reserve 
Officers Training Corps (ROTC). 188 

Student organizing also focused on racist, misogynist, and inequitable cultures within universities, 
tying these to unethical funding practices, and cióse relationships to military and surveillance 
interests. At MIT, gradúate student Arwa Mboya was one of the first to cali for accountability after 
revelations surfaced showing the MIT Media Lab's cióse funding relationship with child sex 
trafficker and assailant Jeffery Epstein. 189 Mboya called on Media Lab Director Joi Ito to resign 
from his position. After investigative journalist Ronan Farrow reported on the ties between Epstein 
and Ito's Media Lab, Ito stepped down. 190 Responding to these disclosures, groups like MIT 
Students Against War organized protests and town halls, demanding that MIT President L. Rafael 
Reif and "all sénior leadership that was aware of this issue" resign. They also demanded a board 
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made up of students, faculty, and staff to review and approve donations. 191 In doing so, they made 
the case that the Epstein revelations were one of many examples of MIT's misogynistic culture, 
pointing to the university's continued employment of undergraduate professor Seth Lloyd, who 
visited Jeffery Epstein in prison and continúes to defend the relationship. 192 

The diverse concerns animating student organizing, and the way in which organizers are tracing 
the interconnections between them, echoes the breadth of focus of the tech-worker movement, 
and a growing recognition that hostile tech cultures are reflected in the technology produced 
within such cultures. 


1.3 Law and Policy Responses 

This year, interest in regulating Al systems increased, with a focus on data protection, algorithmic 
accountability, and biometric/facial-recognition safeguards. Building on the emergence of globally 
oriented data protection approaches such as the European Uniorís General Data Protection 
Regulation (GDPR), policymakers are moving quickly, driven both by the current sense of urgency 
to regúlate the mass deployment of Al technologies lacking discernible safeguards and by the 
failure of ethical frameworks to adequately answer the cali for accountability and justice. 

Data Protection as the Foundation of the Majority of Al Regulatory 
Frameworks 

The relative success of data-protection laws to confront and contain harmful behaviors by 
technology companies provides a natural foundation for approaches to new forms of algorithmic 
activity. 193 In particular, the right to access one's personal data, 194 to access information about 
automated decision-making, 195 and requirements like data protection impact assessments 
(DPIAs) and privacy by design align well with most Al accountability frameworks. 196 As legal 
scholars Margot E. Kaminski and Gianclaudio Malgieri argüe, DPIAs are a bridge between "the two 
faces of the GDPR’s approach to algorithmic accountability: individual rights and systemic 
collaborative governance." 197 

As governments now move to regúlate algorithmic systems, they are not doing so in a policy 
vacuum. Morethan 130 countries 198 have now passed comprehensive data protection laws, with 
Kenya 199 and Brazil 200 being the latest to have modeled their laws largely on the GDPR. While the 
US still lacks a general data protection law, momentum appears to be growing to address this 
gap, with a dramatic increase in activity at both the federal and State levels. 201 

However, there is still an ongoing debate about whether GDPR-style frameworks can or should 
offer a "right to explanation" about specific automated decisions. Some scholars argüe that no 
such right presently exists in the GDPR, 202 while others argüe that múltiple provisions of the GDPR 
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can be pieced together to obtain meaningful information about the logic involved in automated 
decisions. 203 It remains to be seen if this is an effective or even available tool for accountability, as 
there continúes to be a debate over the ways in which transparency 204 and other ways of "seeing 
through data protection laws" can engage with the goals of algorithmic accountability 
frameworks. 

Biometric Recognition Regulation 

This year, numerous regulatory attempts emerged to address the privacy, discrimination, and 
surveillance concerns associated with biometrics—the measurement of unique biological 
characteristics, including data used in facial and affect recognition. These regulatory attempts 
range from bans or moratoriums to laws that would allow the technology on a case-by-case basis 
with specific forms of oversight. 

In Europe, the Swedish government fined a high school for its facial-recognition attendance 
registry as a violation of GDPR 205 France's data protection authority, CNIL, declared it ¡Ilegal to use 
facial recognition in schools based on privacy concerns. 206 

The Australian Parliament took a more aggressive approach, ordering a complete pause on the 
use of a national face database. The moratorium will not be lifted until legislation emerges that 
will allow the government to manage and build the system while acknowledging Citizen digital 
rights and develop a proposal that prioritizes "privacy transparency and ... robust safeguards." 207 
American cities such as San Francisco, Oakland, Seattle, and Somerville similarly have voted to 
ban all forms of government use of the technology 208 

In 2019, members of the United States Congress proposed several biometric bilis, including the 
Commercial Facial Recognition Privacy Act of 2019, 209 the Facial Recognition Technology Warrant 
Act, 210 and the No Biometric Barriers Act of 2019. 211 The latter seeks to prohibit biometric 
recognition in public housing, highlighting many of the same concerns as the tenant organizing at 
Atlantic Plaza Towers in Brownsville, Brooklyn, where residents sought to keep their landlord from 
installing an invasive facial-recognition system in their rent-stabilized apartment complex. 212 

Biometric recognition also emerged this year as a pressing campaign issue. Notably, Democratic 
presidential nominee candidates have publicly taken various stances on the topic, with Senator 
Bernie Sanders adopting the strongest position of calling for a total ban of pólice use of the 
technology. 213 

Meanwhile, several States in the US—Washington, 214 Texas, 215 California, 216 Arkansas, 217 New 
York, 218 and Illinois 219 —have begun actively restricting and regulating in these areas, including 
limits on some forms of biometric collection and recognition. In addition, Washington, 220 
Michigan, 221 California, 222 Massachusetts, 223 Arizona, 224 and Florida 225 have introduced efforts 
seeking to do the same. 


Al Now2019 Report | 33 


Several proposals, such asthe Florida Biometric Privacy Act, the California Consumer PrivacyAct, 
Bill S. 1385 in Massachusetts, NY SB 1203 ¡n New York, and HB1493 in Washington, are explicitly 
modeled after Biometric Information Privacy Act (BIPA), a 2008 Illinois privacy act that serves as a 
high watermark. This is especially true after the Ninth Circuit Court of Appeals approved the 
pursuit of an Illinois class-action lawsuit under BIPA against Facebook's use of facial-recognition 
technology in August, finding that Facebook's collection of biometric face data from users injured 
their rights to privacy. 426 

Key corporate developers of the technology—including Microsoft 227 and Amazon 228 — have also 
come out in support of various forms of regulation on use but have generally resisted 229 calis for 
bans or moratoriums. This strategy mirrors the historie approaches tech companies have taken 
to data protection and other regulatory frameworks that emphasize production pathways and 
compliance over regulatory approach, oversight, and intervention. 

Internal corporate conversations have also addressed regulation. Amazon attempted to block a 
shareholder vote on pausing the company's sale of facial-recognition technology until a 
third-party confirmation that "¡t does not cause or contribute to actual or potential violations of 
human rights." Even after the vote was allowed to go forward by the Securities and Exchange 
Commission (SEC), 230 Amazon aggressively campaigned for shareholders to vote against the 
ban. 231 Body camera manufacturer Axon, 232 in contrast, has adopted an internal ban policy. It 
remains to be seen what concrete impact these efforts will have. 

As the biometric-recognition industry moves full speed ahead with massive investments in 
production and deployment, governments are adopting the technology at a faster rate than they 
are regulating it. France has announced plans to establish a national facial-recognition 
database. 233 In the UK, pólice in Cardiff and London both began trial use of facial-recognition 
technology, leading to legal challenges and objections by civil society groups, academics, and at 
least one department's ethics committee. 234 This year, news of China’s use of biometric 
recognition as weapons of State power to target a Muslim minority 235 and Flong Kong 
protestors 236 made international headlines. In Flong Kong, such surveillance violated their own 
GDPR-style Personal Data (Privacy) Ordinance (PDPO), 237 but China declared a State of 
emergeney and overrode it. 

Algorithmic Accountability and Impact Assessments 

This year, algorithmic accountability bilis profilerated, especially in the United States. As noted 
above, US lawmakers introduced the AAA, which would authorize the Federal Trade Commission 
(FTC) to assess whether corporate automated decisión systems (ADS) producís are biased, 
discriminatory, or pose a privacy risk to consumers. It also requires ADS vendors to submit 
impact assessments to the FTC for evaluation. 
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As Al Now's 2018 report highlighted, 238 the use of algorithmic impact assessments (AlAs) has 
been gaining traction in both policy circles and various countries, States, and cities. 239 Built on the 
success of data-protection, environmental, human-rights, and privacy-impact assessments, AlAs 
require Al vendors and their customers to understand and assess the social implications of their 
technologies before they are used to impact people's Uves. As we outline in our AIA framework, 240 
these assessments would be made publicly available for comment by interested individuáis and 
communities as well as researchers, policymakers, and advocates to ensure they are safe for 
deployment and that those who make and use them are acting responsibly. 

For example, Canada's implementation of AlAs appears under its Directive on Automated 
Decision-Making, as part of the Pan-Canadian Al Strategy, 241 where the Department of Treasury 
embeds the tool into their government procurement process. Australia's Al Ethics Framework also 
contemplates the use of AlAs. 242 Washington became the first State to propose AlAs for 
government ADS with its Plouse and Senate bilis HB 1 65 243 and SB 5527. 244 In addition, some 
scholars have also advocated for a model AIA to complement DPIAs under the GDPR. 245 

Another dimensión to this year's algorithmic accountability legislation was algorithmic 
transparency. As law enforcement agencies increasingly turn to proprietary technology in criminal 
proceedings, the intellectual-property rights of prívate companies are being pitted against 
defendants' right to access information about that technology in order to challenge it in court. 
Addressing the specific case of forensic algorithms like automated software used to analyze DNA 
and predict potential suspects, the Justice in Forensic Algorithms Act of 2019 246 prohibits 
companies from withholding information about their system, such as its source code, from a 
defendant in a criminal proceeding on trade-secrecy grounds. 

Experimentation with Task Forces 

Technologies like predictive analytics and ADS present a number of risks and concerns, especially 
when used by government agencies to make sensitive determinations around who receives 
benefits, which school a child attends, and who is released from jail. Recognizing these risks, 
governments at all levels have begun working to address these concerns, and developing 
governance and accountability mechanisms. 

Of the current approaches, the most common has been the creation of temporary, 
quasi-government bodies (e.g., commissions or task forces), which inelude both external experts 
and government workers. These bodies are tasked with examining emerging technologies and 
publishing their findings, along with recommendations for how ADS systems should be held 
accountable. 

To date, this approach has primarily been implemented by jurisdictions in the United States. 
Alabama, New York City, and Vermont have already commenced their respective commissions 
and task forces, and legislation seeking to create similar bodies is pending in Massachusetts, 
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Washington, and New York State. 247 This follows a tradition in the US, in which task torces and 
similar bodies convene when the government is facing emerging or controversia! issues. With the 
credibility offered by non-governmental experts, task torces (and similar) develop new strategies, 
policies, standards, or guidance that can inform future legislation or regulation.The New York City 
Automated Decisión Systems Task Forcé was the tirst of these quasi-government bodies to 
complete its mándate; however, the process revealed missed opportunities in New York City that 
should be avoided in Vermont, Alabama, and other jurisdictions considering quasi-government 
bodies as a policy intervention. 

The NYC Task Force's shortcomings inelude a signiticant lack of public engagement and the city's 
central role in drafting the NYC ADS Task Forcé Report, which produced a document that did not 
reflect Task Forcé consensus, and was biased in favor of city ADS use. In addition, the law 
enforcement carveout, which was reflected in both the report and an Executive Order issued by 
the Mayor's office, presents a significant omission. Law enforcement's use of automated decisión 
systems, from facial recognition to predictive policing and beyond, poses some of the greatest 
threats to residents, and must be included in any oversight of automated decisión systems. On 
November 19, 2019, Mayor Bill de Blasio released the ADS Task Forcé Report along with an 
executive order to establish an Algorithms Management and Policy Officer within the Mayor's 
Office of Operations. 248 On November 26, 2019, Council Member Peter Koo introduced legislation 
that requires annual reporting on ADS used by city agencies. A more detailed account of the 
missed opportunities and lessons learned from the NYC ADS Task Forcé Process, in addition to 
recommendations for other jurisdictions, can be found in Confronting Black Boxes: A Shadow 
Report ofthe New York City Automated Decisión System Task Forcé. 249 

Litigation Is Filling Some ofthe Void 

This year, various coalitions also continued their attempts to use litigation to hold both 
governments and vendors accountable for harmful uses of Al. 250 

For example, Disability Rights Oregon (DRO) sued the state's Department of Fluman Services over 
sudden cuts in Oregonians' disability benefits with no notice or explanation. In the investigation 
and litigation process, DRO discovered that the reduction was due to the State hard-coding a 
30-percent across-the-board reduction of hours into their algorithmic assessment tool. The State 
quickly accepted a preliminary injunction that restored all recipients' hours to their prior levels, and 
agreed to use the previous versión of the assessment tool going forward. Yet, much like previous 
cases in Idaho and Arkansas, 251 although the Oregon injunction put that particular Al system out 
of Service, it is unclear exactly what the State will offer in its place, and how they will implement it. 

In Michigan, a group of unemployment beneficiaries brought a class-action lawsuit against the 
Michigan Unemployment Insurance Agency (UIA) over a failed automation project, called MiDAS, 
that claimed to be able to detect and "robo-adjudicate" claims of benefits fraud algorithmically. 
The State hired third-party tech vendors to build the system, requesting they design it to 
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automatically treat any data discrepancias or inconsistencies in an individual's record as evidence 
of ¡Ilegal conduct. Between October 2013 and August 2015, the system falsely identified more 
than 40,000 Michigan residents of suspected fraud. The consequences were severe: seizure of 
tax refunds, garnishment of wages, and imposition of civil penalties-four times the amount 
people were accused of owing. And although individuáis had 30 days to appeal, that process was 
also flawed. 

These events prompted a class-action lawsuit filed in State court in 2015 alleging due-process 
violations. After a lower court decisión denied the claim, the Michigan Supreme Court reversed in 
2019 to allow the case to proceed to trial. In the meantime, Michigan continúes to use MiDAS, and 
claims that adjudications are no longer fully automated. It is unclear what (if any) changes were 
made, and whetherthere is any meaningful human review or oversight. 


2. EMERGING AND URGENT CONCERNS IN 2019 

2.7 The Prívate Automation of Public Infrastructure 

As attention to the concerns about Al infrastructures increases, we tend to see them discussed in 
terms of a dichotomy between public and prívate uses. This separation has always been false on 
some level and this year we have seen signs of its eventual collapse, with clear evidence of 
ongoing and expansive integration of public and prívate systems across many different Al 
domains. 

Al and Neighborhood Surveillance 

Troubling partnerships between government and prívate tech companies also emerged as a trend 
this year, especially those that extended surveillance from public environments into prívate 
spaces like prívate properties and the home. 

For example, this summer, a Canadian RCMP troop in Red Deer, Alberta, launched a program 
called CAPTURE to enable "community assisted policing through the use of recorded evidence." 252 
The idea was for commercial businesses and personal residences with prívate security-camera 
infrastructure to effectively share the captured information on their prívate property with the 
pólice, underthe guise of improved community safety. As of November, more than 160 properties 
are participating, effectively covering the entire map of the city, and providing access to the pólice 
surveillance of spaces previously inaccessible without a warrant and consent for entry. 253 Since 
2016, Project Green Light in the City of Detroit in the United States has been working in an almost 
identical fashion. By March of 2019, the mayor of Detroit decided to establish the "Neighborhood 
Real-Time Intelligence Program," described as "a $9 million, State- and federally-funded initiative 
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that would not only expand Project Green Light by installing surveillance equipment at 500 Detroit 
intersections-on top of the over 500 already installed at businesses-but also utilize facial 
recognition software to identify potential crimináis." 254 

Amazon exemplified this new wave of commercial surveillance tech with Ring, a 
smart-security-device company acquired by Amazon in 2018. The central product is its video 
doorbell, which allows Ring users to see, talk to, and record those who come to their doorsteps. 
This is paired with a neighborhood watch app called "Neighbors,” which allows users to post 
instances of crime or safety issues in their community and comment with additional information, 
including photos and videos. 255 A series of reports reveáis that Amazon had negotiated Ring 
video-sharing partnerships with more than 700 pólice departments across the US. Partnerships 
give pólice a direct portal through which to request videos from Ring users in the event of a 
nearby crime investigation. 256 Not only is Amazon encouraging pólice departments to use and 
market Ring producís by providing discounts, but it also coaches pólice on how to successfully 
request surveillance footage from Neighbors through their special portal. 257 As Chris Gilliard, a 
professor who studies digital redlining and discriminatory practices, comments: "Amazon is 
essentially coaching pólice on ... how to do their jobs, and ... how to promote Ring producís." 258 

Neighbors is joined by other apps like Nextdoor and Citizen, which allow users to view local crime 
in real time and discuss it with one another. Ring, Nextdoor, and Citizen have all been criticized for 
feeding into existing biases around who is likely to commit crime; Nextdoor even changed its 
software and policies given extensive evidence of racial stereotyping on its platform. 259 Others see 
these app-based surveillance operations sowing a climate of fear, while tech companies profit 
from a false perception that crime is on the rise. 260 

Smart Cities 

Concerns about the privatization of public space took center stage this year in the debate around 
"smart cities" (municipalities that use data, sensors, and algorithms to manage resources and 
Services). 

Most smart-city initiatives rely on public-private partnerships and technology developed and 
controlled by tech companies, which shifts public resources and control over municipal 
infrastructure and valúes to these companies. 261 Recent research has exposed the extent to 
which majortech companies such as IBM and Cisco have been "selling smartness" by 
disseminating narratives about urban challenges and technological Solutions to those 
challenges. 262 The Alphabet company Sidewalk Labs has similarly been producing visión 
documents replete with renderings of utopian urban scenes. 263 These companies see the 
potential for massive profits: one report estimated the global smart cities market being worth 
$237.6 billion by 2025. 264 
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Smart-city projects around the United States and the world increasingly consolídate power in the 
hands of for-profit technology companies, while depriving municipalities and their residents of 
resources and privacy. The highest-profile example is in Toronto, the home of Sidewalk Lab's 
project to develop "the world's first neighbourhood built from the internet up." 265 A report in 
February 2019 found that Sidewalk Labs has expressed a desire to receive a portlon of the 
property taxes and development fees (estimated at $30 billion over 30 years) associated with the 
project, which would otherwise go to the City of Toronto. 266 And in June 2019, Sidewalk Labs 
released a Master Innovatlon and Development Plan (MIDP), describlng plans to develop and 
manage a far larger plot of land than the 12 acres for which the company was initially given 
license to develop plans. 267 

Smart-city projects also lack transparency and genuine forms of civic participation. 268 Sidewalk 
Labs's civic engagement efforts have been described as a process of obfuscation and 
"gaslighting." 269 Similarly, a contract between urban-planning software company Replica (a 
Sidewalk Labs spinoff company) and the Portland, Oregon regional transportation planning 
agency provides no public access to Replica's algorithms. 270 Siemens is launching a €600 million 
smart-city neighborhood in Berlín, creating "laboratories in reality" with barely any public meetings 
so far. 271 

Many of these public-private partnerships directly enhance the government's surveillance 
capabilities. Chicago and Detroit have both purchased software enabling them to deploy facial 
recognition in the video feeds from cameras across the cities. 272 Similarly, the multinational 
Chínese tech company Huawei's $1.5 billion project to create smart cities in Africa 273 included a 
project in Nairobi where it installed 1,800 cameras, 200 traffic surveillance systems, and a 
national pólice command center as part of its "Safe City" program 274 Huawei's Safe City 
technology has been used by some African governments to spy on political opponents. 275 

In other cities, behind-the-scenes data-sharing arrangements allow data collected by prívate 
companies to flow into law-enforcement agencies. San Diego has installed thousands of 
microphones and cameras on Street lamps in recent years in an effort to study traffic and parking 
conditions; although the data has proven of little use in improving traffic, the pólice have used the 
video footage in more than 140 cases without any oversight or accountability 276 The City of 
Miami is actively considering a 30-year contract with lllumination Technologies, providing the 
company with free access to set up light poles containing cameras and license-plate readers, 
collecting information that will filter through the Miami Pólice Department (and that the company 
can use in unchecked ways). 277 Documents obtained via public-records requests showed that 300 
pólice departments in California have access, through Palantir, to data collected and stored by the 
Department of Homeland Security's Northern California Regional Intelligence Center, without any 
requirement to disclose their access to this information. 278 

Numerous groups are beginning to push back on the encroaching privatization fueled by smart 
cities, with the most concerted and organized effort in Toronto. In February, a group of 30 
Torontonians launched the #BlockSidewalk campaign, 279 and has noted that the project "is as 


Al Now2019 Report | 39 


much about privatization and corporate control as ¡t ¡s about privacy.'' 280 In April, the Canadian 
Civil Liberties Association (CCLA) filed a lawsuit against Waterfront Toronto, arguing the 
organizaron abused its legal authority in granting Sidewalk Labs the authority to develop 
data-governance policy 281 And after Sidewalk Labs released its MIDP, the Chairman of Waterfront 
Toronto (the government task forcé charged with managing the Sidewalk Labs project) critiqued 
the proposal in a public letter as "premature." 282 

By the end of October, Waterfront Toronto had reached a new agreement with Sidewalk Labs, 
restricting Sidewalk Labs to the original 12-acre parcel and asserting the government's role as 
leading key components of the project. 283 The project's ultímate fate is still undetermined: 
Waterfront Toronto continúes to review the project and will come to a final decisión about 
whetherto proceed by March 31, 2020 284 

Al at the Border 

Al continúes to play a larger and more pernicious role in the targeting of immigrant populations 
within the United States. 

The increasing use of Al technologies is often justified based on nationalist rhetoric. In the US, 
talk of a "smart wall" that utilizes drones, sensors, and increased facial recognition to detect 
individuáis is receiving bipartisan support in design and ¡mplementation. 285 Anduril Industries, a 
technology company that recently replaced Google on a Project Maven Department of Defense 
contract developing Al-based surveillance systems and that also produces autonomous 
drones, 286 now provides solar-powered "sentry" towers for the Customs and Border Protection 
(CBP) agency. 287 One of Anduril's earliest investors, Peter Thiel, also founded the company 
Palantir Technologies, which provides database management and Al to ICE. Palantir's tech has 
enabled agencies such as ICE to combine and analyze information from varying government 
databases, and to use this to track, target, and detain people whom they believe are in the US 
"illegally. ,,288 In July, the Washington Post reported on thousands of internal documents and emails 
obtained through public-records requests by researchers at Georgetown Law's Center on Privacy 
and Technology. The documents showed that the Federal Bureau of Investigation (FBI) and ICE 
were using State driver's license databases as "the bedrock of an unprecedented surveillance 
infrastructure” that relied on facial-recognition technology. 289 The US Justice Department also 
recently announced plans to collect DNA data from migrants Crossing the border, which could 
create more invasive monitoring of immigrants without any real limits. 290 

Outside the US, governments are equally eager to pilot Al systems at border checkpoints. The EU 
aims to deploy an Al-based "lie detector" built by ¡BorderCtrl, but makes no mention of the 
predictive accuracy or the inherent bias that might exist within such tools. 291 In the UK, the Home 
Office facial-recognition systems were found to be wrongfully identifying travelers as crimináis, 
delaying theirtravels and detaining them with no elements of due process. 292 Meanwhile, some 
US-based technology companies have been found to be deploying these faulty security Solutions 
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outside of the US. Journalists recently revealed that Microsoft funded an Israeli firm to conduct 
facial-recognition surveillance on West Bank Palestinians in public space. 293 China, having already 
built massive surveillance capital to track and identify citizens anywhere in the country and 
beyond, now also employs affect recognition to try to identify crimináis at airports and subway 
stations 294 

The significant growth in Al use for bordertracking, surveillance, and prediction threatens the 
rights and civil liberties of residents within a country's borders, not only those outside of it. In the 
US, such technologies act as forcé multipliers for ICE and CBP, amplifying their ability to track and 
target immigrants and residents. Such powers are extended well beyond the border between the 
US and México, and often do so at the expense of constitutional rights. The US border zone 
consists of a hundred-mile band, from the border inland, tracing the whole of the US. Nearly 
two-thirds of US residents live within it. As the ACLU says in their analysis of the problems with 
CBP's authority within the US border zone: "The spread of border-related powers inland is 
inseparable from the broader expansión of government intrusión in the Uves of ordinary 
Americans." 295 And as Al-enabled surveillance, tracking, and targeting in the context of border 
security become more pervasive and more powerful, such technologies will inevitably be used on 
US residents, further infringing on rights and liberties within the country. 

National Biometric Identity Systems 

An increasing number of governments across the world are building national biometric identity 
systems that generate a unique identifier for each person, typically serving as a link to discrete 
government databases. Residents in many countries are increasingly required to use these new 
digital modes in orderto access a range of Services. Along with demographic information, 
biometrics like fingerprints, iris scans, or facial scans are used either for one-time enrollment into 
an ID database or as a continuing means of authentication. These ID systems vary in terms of 
whom they are meant to inelude (and exelude): residents, citizens, or refugees. 296 Many of these 
projeets are in countries in the global South and have been actively encouraged as a development 
priority by organizations like the World Bank under the "ID4D” banner 297 and supported as fulfilling 
the UN Sustainable Development Goals. 298 Although these projeets are often justified as creating 
efficiencies in the rollout of government Services to benefit the "end user," they appear to more 
directly benefit a complex mix of State and prívate interests. 

India, for example, introduced a national ID to supposedly create more efficient welfare 
distribution that also happened to be designed for market activity and commercial surveillance. 299 
Until intervention by the Indian Supreme Court, 300 any prívate entity was allowed to use the state's 
biometric ID infrastructure for authentication, including banks, telecom companies, and a range of 
other prívate vendors with little scrutiny or privacy safeguards. A recent report 301 describes how ID 
databases in Ghana, Rwanda, Tunisia, Uganda and Zimbabwe are facilitating "Citizen scoring" 
exercises like credit reference bureaus to emerge at scale. 
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The ¡nvolvement of foreign technology vendors for key technical functions has also raised serious 
national-security concerns in Kenya 302 and India. There have already been múltiple attempts at 
breaching these ID databases, 303 and there was a security flaw in the Estonian ID system, which 
was otherwise celebrated as a technically advanced and privacy-respecting model. 304 A security 
breach of the biometrics in these databases could potentially create lifelong impacts forthose 
whose bodily information is compromised. 

The dossiers of authentication records created by these ID systems, as well as the ability to 
aggregate information across databases, can increase the power of surveillance infrastructures 
availableto governments. Kenya's Home Minister referred to its recently announced biometric ID 
system "Huduma Numba" as creating a "single source of truth" about each Citizen . 305 

Enrollment and associated data collection for these ID systems has been coercive because it is 
either de facto or legally mandatory to be enrolled to access essential Services. These instances 
must be understood against the backdrop of claims that these systems will create cost savings 
by weeding out fake or "ghost" 306 beneficiaries of welfare Services, which replays the familiar logic 
of using technical systems as a way to implement and justify austerity policies. 307 In India and 
Perú, múltiple cases of welfare-benefits denials led to higher malnutrition levels 308 and even 
starvation deaths 309 because people either were not enrolled or were unable to authenticate due 
to technical failures. 

There is growing concern about the assumed efficiency of these automated systems, as well as 
about whom these technical systems benefit and at what cost. This year, the Jamaican Supreme 
Court struck down 310 Jamaica's centralized, mandatory biometric ID system, noting that the 
project led to privacy concerns that were "not justifiable in a free and democratic society.” Soon 
after, Ireland's Data Protection Commissioner ordered the government to delete the ID records of 
3.2 million people after it was discovered that the new "Public Services Card" was being used 
without limits on data retention or sharing between government departments. 311 After years of 
civil society protest and strategic litigation against the Indian biometric ID system Aadhaar, the 
Indian Supreme Court put several limits on the use of the system by prívate companies (although 
it has permitted large-scale and coercive government use). 312 The Kenyan Supreme Court is 
currently hearing múltiple constitutional challenges to Huduma Namba, the national ID system 
that proposes to collect a range of biometrics including facial recognition, voice samples, and 
DNAdata. 313 

These moments of backlash have not deterred other governments and donor agencies from 
pushing similar centralized biometric ID systems elsewhere. Just this October, the Brazilian 
government announced its intention to create a centralized Citizen database for every resident, 
¡nvolving the collection of a wide range of personal information, including biometrics. 314 France 
has announced that it will trial facial scans to enroll citizens in its latest national ID venture. 315 

As these projects continué to emerge across the world, more research into the international 
political economy of these ID systems is required. Civil society coalitions like the #WhylD 
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campaign 316 are coming togetherto fundamentally question the ¡nterests driving these projects 
nationally and through international development organizations, as well as developing advocacy 
strategies to influence their development. 317 

China Al Arms Race Narrative 

ln the last couple of years, the "Al arms race" between the US and China (and to a lesser extent 
Russia) has become a frequent topic of publlc discourse. 318 This "race" is commonly cited as a 
reason the US and the tech companies that produce the country's Al systems need to ramp up Al 
development and deployment and push back against calis for slower, more intentional 
development and stronger regulatory protections. 

It is ¡mportant to question the role that the "Al arms race” narrative plays in the discourse around 
Al. Who is driving it, and what ¡nterests and power structures do they represent? Critically, what 
futures do they guide us toward? And on what basis are these claims founded, and how is 
"progress” in such a race measured? 

Metrics comparing US and China Al development often focus on the proportion of top Al 
scientists and engineers who reside and work in each country, whether Chínese or US researchers 
authored the most cited technical papers, or how many Al patents emerged from each country. 319 
Based on such evidence, recent studies have warned that China could "overtake” the United 
States in this measure by 2020, 320 with others warning that top Al scientists from Silicon Valley 
are emigrating to China to join competing Chínese companies. 321 

While China and the US are certainly leading based on these measures of technical Al 
development, with profound geopolitical implications, it's also important to look at what these 
measures omit. Empirical factors like where Al produced in either country will eventually be 
deployed, what purpose such systems will be put to, whether they work, and which communities 
bear the risk of bias and other harms are rarely discussed within the "arms race” discourse. Given 
the mounting evidence of harm due to Al systems being applied in sensitive social contexts, these 
questions are urgent. And yet they are not being considered in the current estimation of which 
country is "winning" the "Al race." Asking such questions can help assess the goals of "the arms 
race" itself, and what the implications of "winning" the race might be. 

Proponents of the Al arms race narrative also tend to measure "progress" based on Al-industry 
cooperation with the military establishment, characterizing the reticence of those who would 
question the development of weapons systems and mass surveillance systems as implicitly 
"anti-progress" or unpatriotic. Indeed, some of the most consistent voices warning of the dangers 
of Chínese Al supremacy come from within the US defense establishment. 322 This fits with the 
growing attention to the military use of Al, and the cali for increased military spending on Al and 
closer partnerships between the US military and Silicon Valley. 323 
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Chinese tech companies' purported willingness to work on weapons and military technology ¡s 
frequently contrasted with the US, where tech workers, human rights groups, and academics have 
pushed back against Silicon Valley companies entering into contraéis with US military efforts 
(such as opposition within Google to Project Maven). 324 Such resistance to privatized, Al-enabled 
weapons and infrastructure is seen as causing unjustified friction in this race. 325 Former Secretary 
of Defense Ashton Cárter said that it was "ironic" that US companies would not be willing to 
cooperate with the US military, "which is far more transparent [than the Chinese] and which 
reflects the valúes of our society." 326 

More broadly, this view of progress tends to see all calis for restraint, reflection, and regulation as 
a strategic disadvantage to US national interest. It turns accountability into a barrier to progress 
and suppresses calis for oversight. 327 At a time when "move fast and break things" is 
acknowledged to have done long-term harm to core social and political infrastructures, this 
emphasis on speed seems particularly misplaced. 

The urgency of "beating" China is commonly justified based on the nationalist assumption that 
the US would imbue its Al technologies, and its application of said technologies, with better 
valúes than China would. China's authoritarian government is presumed to promote a more 
dystopian technological futurethan Western liberal democracies. 

The Chinese government's oppression of minorities through state-private partnerships (including 
a significant reliance on US technology) is well documented and rightly condemned by human 
rights organizations. 328 Yet, China's use of Al should serve as a warning to the US of what can 
happen when you puf technological "progress” above human rights and civil liberties. And there is 
growing evidence that the US is increasingly using Al in oppressive and harmful ways that echo 
China's use. Scholars like Shazeda Ahmed 329 and Abeba Birhane 330 point out that many elements 
of Chinese Al technology are actually commonplace in the US digital economy. However, such 
applications of Al in the US are frequently enabled by prívate Al companies, from Amazon selling 
facial recognition to law enforcement to Palantir providing surveillance and tracking infrastructure 
to ICE. Such uses are often protected by contractual secrecy, and not disclosed as State policy. 
And when they are exposed, it's generally by whistleblowers and investigative journalists, not by 
the companies or agencies partnering to develop and apply these Al systems. 


2.2 From "Data Colonialism " to Colonial Data 

The Abstraction of "Data Colonialism" and Context Erasure 

"Data colonialism" and "digital colonialism” have become popular metaphors for academics, 331 
policymakers, and advocacy organizations looking to critique harmful Al practices. In these 
accounts, colonialism is generally used to explain the extractive and exploitative nature of the 
relationship between technology companies and people, deployed toward varying political ends. 
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ln Europe, for example, ¡t is used by advocacy groups to argüe for a movement toward "digital 
sovereignty" that encourages decentralized and community-owned data-governance 
mechanisms. 332 In India, domestic industrialists and policymakers have argued that Silicon Valley 
tech giants are "data colonizers” and that national companies, ratherthan foreign ones, must get 
first priority accessing Indians' data. 333 

However, using data colonialism as a metaphor to abstract tech company practices overlooks 
specific historical, cultural, and political contexts and obscures the fact that present-day Al labor 
and economic structures exist because of actual histories of colonization. 334 Growing research on 
the locally specific real-world impact of the Al industry on countries in the global South makes 
visible these contexts and the lived human conditions behind the technology and data. 335 As 
demonstrated by policy narratives in India, abstracting "colonialism” can allow narrow economic 
interests to co-opt the rhetoric of decolonial struggles while replicating the same extractivist 
logics of their Silicon Valley counterparts. This has led to a growing critique of these metaphorical 
usages, 336 and a need to recenterthe narrative on lived experiences to build broader solidarity with 
communities directly affected by Al. 

Colonial Data: Statistics and Indigenous Data Sovereignty 

Indigenous communities have been at the forefront of resisting harms caused by data 
abstraction. 337 For example, advocacy groups have drawn attention to the ways that census 
information and population counts function 338 as a core feature of settler-colonial governance, 
feeding massive amounts of abstracted data into digital systems. 339 Problematic uses of such 
"Indigenous statistics” in census administration directly linkto underrepresentation and the lack 
of resources these communities face. 340 

In the context of open-data movements, a number of Indigenous-led movements for sovereignty 
and self-determination over data and data analysis have emerged. The term "Indigenous data 
sovereignty” (ID-Sov) is generally defined as "the right of a nation to govern the collection, 
ownership, and application of its own data." 341 The term data sovereignty, like data colonialism, is 
currently used by both Indigenous and non-lndigenous policy and advocacy groups to make 
appeals in data ownership and proprietary rights, but with very different historical, social, and 
political contexts. 342 

These groups have implemented new programs, organizational frameworks, and data policy to 
address Indigenous data sovereignty and data governance across local, national, and 
transnational contexts. In 2016, the US Indigenous Data Sovereignty NetWork (USIDSN) was 
established to "link American Indian, Alaska Native, and Native Hawaiian data users, tribal leaders, 
information and communication technology providers, researchers, policymakers and planners, 
businesses, Service providers, and community advocates together to share stories about data 
initiatives, successes, and challenges, and resources.” The same year, a collective of Máori 
scholars and government leaders and Aboriginal rights developers published the book Indigenous 
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Data Sovereignty: Toward an Agenda ¡n response to oversights in the United Nations Declaration 
on the Rights of Indigenous Peoples (UNDRIP). 343 

The Indigenous Data Sovereignty program set forth to address "the twin problems of a lack of 
reliable data and information on indigenous peoples and biopiracy and misuse of their traditional 
knowledge and cultural heritage." 344 

Advocacy groups are establishing sovereignty and ownership protocols at the level of data and 
analysis. 345 For example, the Local Contexts initiative aims to support Native, First Nations, 
Aboriginal, Inuit, Metis, and Indigenous communities in the management of their intellectual 
property and cultural heritage in the growing digital environment. 346 Their Traditional Knowledge 
or TK labels are "designed as a tool for Indigenous communities to add existing local protocols for 
access and use to recorded cultural heritage that is digitally circulating outside community 
contexts." 347 TK labels are a framework for labeling data through local decisión and preserved in 
circulation and exchange. The US Library of Congress has recently integrated TK labels to digitally 
reformat older media formats "to recover and preserve the recorded voices and languages of 
Native American people." 348 More than an archival process, Local Contexts is working toward "a 
new paradigm of rights and responsibilities that recognizes the inherent sovereignty that 
Indigenous communities have over their cultural heritage." This moves toward the possibility of 
reconfiguring entire information systems, like the US Library of Congress, according to Indigenous 
sovereignty guidelines. 349 

In September, the Global Indigenous Data Alliance (GIDA) launched. 350 Responding directly to the 
international open-data and open-science debates, GIDA has put forward a set of "CARE 
principies” that address the power differentials and historical contexts neglected by the open-data 
movements' "FAIR principies," which valué data as "findable, accessible, interoperable, reusable." 351 
GIDA aims to establish internationally recognized protocols of meaning for local Indigenous data 
and to assert valúes for data generation, circulation, and application beyond the culturally 
flattening notion of open accessibility. GIDA's data CARE principies are Collective benefit, 

Authority to control, Responsibility, and Ethics. 


2.3 Bias Built In 

While Indigenous peoples are calling for data sovereignty and bringing attention to the thorny 
relationships among how individuáis are defined by data and the resources allocated to them, the 
same set of issues has recently played out on the national American stage, prompting 
investigations and calis for reform. 

In November, the prominent software engineer and author David Heinemeier Hansson propelled a 
renewed wave of criticism of algorithmic discrimination with his tweet about the Apple Card. Fie 
castigated Apple because he received a credit limit 20 times higherthan his wife, Jamie 


Al Now2019 Report | 46 


Heinemeier Hansson. 352 Meanwhile, Apple cofounder Steve Wozniak's wife, Janet Hill, was 
allocated a credit limitthat was a mere 10 percent of her husband's. 353 Ultimately, Hansson's 
complaint about the sexist Apple Card algorithms triggered investigations by both the Senate 
Committee on Finance and the New York State Department of Financial Services. 354 

Flansson blamed a sexist black-box algorithm, echoing and amplifying the work of numerous 
activists, journalists, researchers, and tech workers (like Flansson himself) who have been 
warning of the dangers of biased Al systems for at least a decade. 355 Many of those pioneering 
this work are people of color, and they predominantly identify as women or non-binary. They have 
rigorously researched, detected, and proved bias in advertising networks, facial recognition, 
search engines, welfare systems, and even algorithms used in criminal sentencing. 

Flowever, the tech industry-which by contrast is predominantly led by white, wealthy men 356 - 
has been doing its best to resist, minimize, and even mock this critical and alarming work. 
Flansson himself observed the denialism in the responses to his tweet criticizing the Apple Card. 
Fie commented: "Every single póster questioning my wife's credit score, a man. Every single 
defense of Apple blaming G[oldman] S[achs], a man. Almost like men are over represented in the 
defense/justification of discrimination that doesrít affect them?" 357 Indeed, Flansson had 
encapsulated the defensiveness of Google Chairman Eric Schmidt speaking at Stanford 
University late in October; Schmidt groused, "We know about data bias. You can stop yelling about 
it." 358 

Although Schmidt's comment appeared offhand, it served to underscore that tech companies are, 
in fact, deeply aware that algorithmic discrimination is entrenched in the systems with which they 
are blanketing the world. In fact, Microsoft's latest report to shareholders flagged reputational 
harm due to biased Al systems among the company's risks. 359 Although the industry is proffering 
ostensible Solutions such as corporate Al ethics, those Solutions are failing, as discussed 
elsewhere in this report. So far, Big Tech refuses to prioritize solving these issues over their 
bottom line. 

Recent research has highlighted how the bias built into the tech industry works in a feedback loop 
from a dire lack of diversity among employees to damaging discrimination embedded in 
algorithms. 360 One study found that only 18 percent of speakers at leading Al conferences were 
women, 361 while another showed that 80 percent of Al professors are men. 362 Indicators suggest 
that things look much worse when considering representation by race, ethnicity, or ability. 363 Such 
diversity of experience is a fundamental requirement for those who develop Al systems to identify 
and reduce the harms they produce. 364 

Just as the so-called solution of corporate Al ethics is no solution at all, the so-called solution to 
"diversity in tech" often centers simply around highly visible "Women in Tech" initiatives that 
privilege white women above others. 365 But the challenges go far beyond this, from barriers to 
entry to toxic experiences within Al spaces. 366 In fact, a recently released documentary, Losing 
Lena, highlights the five-decade-long history of how an image of a nude woman has continued to 
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be used in the tech industry, pointing to "the many small ways in which women (and ... people of 
color) are told they do not belong in the tech industry." 367 

What would it look like to expand the trame? For one, attending to the experiences of 
communities who have been erased from the field of Al does much more than foster inclusión or 
help identify where technologies produce biased outcomes. It also surfaces new ways of 
understanding how technological systems order our social world. The work of disability scholars 
and activists has much to offer in explaining both the processes and consequences of erasure. 368 
Disability scholarship also emphasizes that the concept of "normal" and the tools and techniques 
for its enforcement, has historically constructed the disabled body and mind as deviant and 
problematic. Scholars in this area may consider how we can better assess the normative models 
encoded in Al systems, and what the consequences of enforcing these norms may be (and for 
whom). 

The integration of community voices and stories at only a surface level is all too often used as a 
marketing strategy, mobilized to support practices that are ultimately not in Service of community 
needs. In attending to affected communities it is crucial that their needs be centered, in their own 
voices. As the credo of the disability rights movement puts it: "Nothing about us, without us." It is 
critical that participation in the work of actively resisting discrimination in the field be seen as the 
responsibility of everyone, and that the labor of resistance not be left solely to those who are 
most affected by the harms of Al. 


2.4 Al and the Climate Crisis 


On September 20, workers from 12 tech companies joined the global climate strike. 369 They 
highlighted tech's role in climate change and demanded "zero carbón emissions by 2030, zero 
contracts with fossil fuel companies, zero funding of climate denial lobbying or other efforts, and 
zero harm to climate refugees and frontline communities." 370 

This might have surprised some people, as tech's contribution to the climate crisis is rarely 
acknowledged. Indeed, industry marketing often highlights green policies, sustainability initiatives, 
and futures in which Al and other advanced technologies provide Solutions to climate problems. 
But the tech sector is a significant contributor to climate change and environmental harms. 371 

Al Makes Tech Dirtier 

The tech industry faces criticism for the significant energy used to power its computing 
infrastructure. As a whole, the industry's energy dependence is on an exponential trajectory, with 
best estimates showing that its 2020 global footprint amounts to 3.0-3.6 percent of global 
greenhouse emissions, more than double what the sector produced in 2007. 372 This is 
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comparable to that of the aviation industry, 373 and larger than that of Japan, which is the fifth 
biggest polluter in the world. 374 In the worst-case scenario, this footprint could increase to 14 
percent of global emissions by 2040. 

In response, the majortech companies have made data centers more efficient, and have worked 
to ensure they're powered at least in part by renewable energy-changes they're not shy about, 
announcing them with marketing blasts and much public fanfare. 375 These changes are a step in 
the right direction, but dorít come cióse to tackling the problem. Most large tech companies 
continué to rely heavily on fossil fuels, and when they do commit to efficiency goals, these are 
most often not open to public scrutiny and validation. 376377 

The Al industry is a significant source of further growth in greenhouse emissions. With the 
emergence of 5G networks aiming to realize the "internet of things," the increased acceleration of 
data collection and traffic is already underway. 378 In addition to 5G antennas consuming far more 
energy than their 4G predecessors, 379 the introduction of 5G is poised to fuel a proliferation of 
carbon-intensive Al technologies, including autonomous driving 380 and telerobotic surgery. 381 

A core contributor to the Al field's growing carbón footprint is a dominant belief that "bigger is 
better." In other words, Al models that leverage massive computational resources to consume 
larger training dataseis are assumed to be inherently "better" and more accurate. 382 Whilethis 
narrative is inherently flawed, 383 its assumptions drive the use of increased computation in the 
development of Al models across the industry. 

Last year, researchers Dario Amodei and Danny Hernández at OpenAl reported that "[s]ince 2012, 
the amount of [computation] used in the largest Al training runs has been increasing exponentially 
with a 3.4 month doubling time (by comparison, Moore's Law had an 18 month doubling 
period)." 384 Their observations show developers "repeatedly finding ways to use more chips in 
parallel, and ... willing to pay the economic cost of doing so”. 

As Al relies on more computers, its carbón footprint increases, with significant consequences. A 
recent study from the University of Massachusetts, Amherst estimated the carbón footprint of 
training a large natural-language Processing model. Emma Strubell and her coauthors reported 
that training just one Al model produced 300,000 kilograms of carbón dioxide emissions. 385 That's 
roughly the equivalent of 125 round-trip flights from New York to Beijing. 

Al and the Fossil Fuel Industry 

Adding to their already sizeable climate impact, big Al companies are aggressively marketing their 
(carbon-intensive) Al Services to oil and gas companies, offering to help optimize and accelerate 
oil production and resource extraction. Amazon is luring potential customers in the oil and gas 
industry 386 with programs like "Predicting the Next Oil Field in Seconds with Machine Leaming." 387 
Microsoft held an event called "Empowering Oil & Gas with Al," 388 and Google Cloud has its own 
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energy vertical dedicated to working with fossil fuel companies. 389 And C3 loT, an Al company 
originally created to facilítate the transition to a society fueled by renewable energy, now helps 
large oil and gas companies, including Royal Dutch Shell, Baker Hughes, and Engie, to expedite 
their extraction of fossil fuel. 390 

A recent article in Logic points out that oil and gas account for 30 percent of the total addressable 
market, making "the success of Big Oil, and the production of fossil fuels ... key to winning the 
cloud race." 391 Recently, the Guardian examined the role of Big Tech in sustaining the market for 
fossil fuel, illuminating the massive amounts of money tech companies invest in organizations 
that actively campaign against climate legislation, and promote climate change denial. 392 

Opacity and Obfuscation 

When researchers and policymakers attempt to account fortech's climate footprint, it is 
immediately clear how little information is available. They are left to rely on voluntary company 
disclosures, without access to the information they would need to make a thorough accounting of 
tech's true energy use. 

There is very little public data available, and few incentives for tech companies to release it. 
Without the information necessary to reach robust conclusions, researchers Lotfi Belkhir and 
Ahmed Elmeligi had to estímate 2018 data-center energy consumption using data from 2008. 393 It 
was all they had to work with, even though, over the past ten years, both the scale of computation 
and the technologies powering it have changed radically. 

The authors of Greenpeace’s report make similar observations, stating that while efficiency 
metrics have been eagerly adopted by the industry, "very few companies report under newer 
metrics ... that could shed any light on the basic question: how much dirty energy is being used, 
and which companies are choosing clean energy to power the cloud?" 394 More frustratingly, the 
unwillingness of cloud providers to provide customers with insight into the energy use of 
procured Services forms a critical barrierto meaningful carbón accounting across all sectors and 
organizations that rely on digital technology. 


2.5 Flawed Scientific Foundations 

Concerns about Al systems focus not only on the harms caused when they are deployed without 
accountability; they also include the underlying and often flawed scientific foundations upon 
which they are built and then marketed to the public. This year, researchers uncovered systems in 
wide deployment that purport to operationalize proven scientific theories, but in the end are little 
more than speculation. 395 This trend in Al development is a growing area of concern, especially as 
applied to facial- and affect-recognition technology. 
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Facial/Affect Recognition 

Affect recognition is an Al-driven technology that claims to be able to detect an individual's 
emotional State based on the use of computer-vision algorithms to analyze their facial 
microexpressions, tone of voice, or even their gait. It is rapidly being commercialized for a wide 
range of purposes-from attempts to identify the perfect employee 396 to assessing patient pain 397 
to tracking which students are being attentive in class. 398 Yet despite the technology's broad 
application, research shows affect recognition is built on markedly shaky foundations. 

The affect-recognition industry is undergoing a period of significant growth: some reports indícate 
that the emotion-detection and -recognition market was worth $12 billion in 2018, and by one 
enthusiastic estímate, the industry is projected to grow to over $90 billion by 2024. 399 These 
technologies are often layered on top of facial-recognition systems as a "valué add." 

For example, the company Kairos is marketing video-analytics cameras that claim to detect faces 
and then classify them as feeling anger, fear, and sadness, along with collecting customer identify 
and demographic data. Kairos sells these producís to casinos, restaurants, retail merchants, real 
estáte brokers, and the hospitality industry, all with the promise that they will help those 
businesses see inside the emotional landscape of their patrons. 400 In August, Amazon claimed its 
Rekognition facial recognition software could now assess fear in addition to seven other 
emotions. Though it declined to provide any details on how it is being used by customers, it 
indicated retail as a potential use case, illustrating how stores can feed live images of shoppers to 
detect emotional and demographic trends. 401 

Employment has also experienced a surge in the use of affect recognition, with companies like 
HireVue and VCV offering to screen job candidates for qualities like "grit" and to track how often 
they smile. 402 Cali center programs Cogito and Empath use voice-analysis algorithms to monitor 
the reactions of customers and signal to cali agents when they sound distressed 403 Similar 
programs have been proposed as an assistive technology for people with autism, 404 while 
Boston-based company BrainCo is creating headbands that purport to detect and quantify 
students' attention levels through brain-activity detection, 405 despite studies that outline 
significant risks associated with the deployment of emotional Al in the classroom 406 

Affect-recognition software has also joined risk assessment as a tool in criminal justice. For 
example, pólice in the US and UK are using the eye-detection software Converus, which examines 
eye movements and changes in pupil size to flag potential deception. 407 Oxygen Forensics, which 
sells data-extraction tools to clients including the FBI, Interpol, London Metropolitan Pólice, and 
Flong Kong Customs, announced in July it also added facial recognition, including emotion 
detection, to its software, which ineludes "analysis of videos and images captured by drones used 
to identify possible known terrorists" 408 
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But often the software doesn't work. For example, ProPublica reported that schools, prisons, 
banks, and hospitals have installed microphones from companies that carry software developed 
by the company Sound Intelligence, purporting to detect stress and aggression before violence 
erupts. But the "aggression detector" was not very reliable, detecting rough, higher-pitched sounds 
like coughing as aggression. 409 Another study by researcher Dr. Lauren Rhue found systematic 
racial biases in two well-known emotion-recognition programs: when she ran Face++ and 
Microsoft's Face API on a dataset of 400 NBA player photos, she found that both systems 
assigned black players more negative emotional scores on average, no matter how much they 
smiled. 410 

There remains little to no evidence that these new affect-recognition producís have any scientific 
validity. In February, researchers at Berkeley found that in orderto detect emotions with accuracy 
and high agreement requires context beyond the face and body. 411 Researcher Rubén van de Ven 
makes this point in his exploration of affect recognition, citing the "Kuleshov Effect," an 
experiment from the beginning of thetwentieth century in which filmmaker Lev Kuleshov "edited 
three video sequences. Each sequence showed the same 'neutral' face of a man, followed by the 

image of a dead man, a píate of soup or a woman_When these sequences were shown, the 

audience 'raved about the acting', believing the man who 'looked' at the dead man, the soup or the 
woman, was either expressing grief, hunger or desire." 412 Others at the University of Southern 
California called for a pause in the use of some emotion analytics techniques at the 8th 
International Conference on Affective Computing and Intelligent Interaction this year. "'[T]his facial 
expression recognition technology is picking up on something - it's just not very well correlated 
with what people want to use it for. So they're just going to be making errors, and in some cases, 
those errors cause harm,'” said Professor Jonathan Gratch 413 

A major review released this summer found that efforts to "read out" people's internal States from 
an analysis of facial movements alone, without considering context, are at best incomplete and at 
worst entirely lack validity. 414 After reviewing over a thousand studies on emotion expression, the 
authors found that, although these technologies claim to detect emotional State, they actually 
achieve a much more modest outcome: detecting facial movements. As the study shows, there is 
a substantial amount of variance in how people communicate their emotional State across 
cultures, situations, and even across people within a single situation. Moreover, the same 
combination of facial movements-a smile or a scowl, for ¡nstance-can express more than a 
single emotion. The authors conclude that "no matter how sophisticated the computational 
algorithms ... it is premature to use this technology to reach conclusions about what people feel 
on the basis of their facial movements.” 415 

Given the high-stakes contexts in which affect-recognition systems are being used and their rapid 
proliferation over the past several years, their scientific validity is an area in particular need of 
research and policy attention-especially when current scientific evidence suggests that claims 
being made about their efficacy don't hold up. In short, we need to scrutinize why entities are 
using faulty technology to make assessments about character on the basis of physical 


Al Now2019 Report | 52 


appearance in the first place. This is particularly concerning ¡n contexts such as employment, 
education, and criminal justice. 

Face Datasets 

Following the release of several studies, there continué to be significant performance disparities 
in commercial facial-recognition producís across intersectional demographic subgroups. 416 In 
response, some companies are trying to "diversify" datasets to reduce bias. For instance, 
computer-vision company Clarifai revealed that it makes use of the profile photos from the dating 
website OkCupid to build large and "diverse" datasets of faces 417 Clarifai claims the company 
gave them explicit permission and access to the data, so it remains unclearto what extent such 
data brokering constitutes a legal privacy violation disproportionately affecting people of color. 
IBM undertook a similar endeavor after being audited, releasing its "Diversity in Faces” study, 
which included an "inclusive" dataset of faces from a wide variety of Flickr users 418 Although 
most of the users whose images were harvested had given permissions under an open Creative 
Commons license, 419 enabling widespread Internet use, none of the people in the photos gave IBM 
permission, again raising serious legal and ethical concerns about such practices. 420 

The problematic practice of scraping online images to produce diverse datasets is not limited to 
industry alone. Researchers Adam Flarvey and Jules LaPlace exposed similar methods used to 
collect faces for academic dataseis 421 Most notably, the DUKE MTMC dataset, 422 Brainwash 
dataset, 423 and others 424 were collected by setting up surveillance cameras at college campuses, 
detecting and cropping out the faces of unsuspecting students to add to their database. 

Ultimately, simply "diversifying the dataset" is far from sufficient to quell concerns about the use 
of facial-recognition technology. In fact, the face datasets themselves are a collection of artifacts 
to uncover, the assemblage of which reveáis a set of decisions that were made regarding whom 
to inelude and whom to omit, but more importantly whom to exploit. It will be essential to 
continué to tell these stories, and to begin to uncover and perhaps challenge our accepted 
practices in the field, and the problematic patterns they reveal. 425 


2.6 Health 

Al technologies today medíate people's experiences of health in many ways: from popular 
consumer-based technologies like Fitbits and the Apple Watch, to automated diagnostic support 
systems in hospitals, to the use of predictive analytics on social-media platforms to predict 
self-harming behaviors. Al also plays a role in how health insurance companies generate 
health-risk scores and in the ways government agencies and healthcare organizations allocate 
medical resources. 426 
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Much of this activity comes with the aim of improving people's health and well-being through 
increased personalization of health, new forms of engagement, and clinical efficiency, popularly 
characterizing Al in health as an example of "Al for good" and an opportunity to tackle global 
health challenges. 427 This appeals to concerns about Information complexities of biomedicine, 
populatlon-based health needs, and the rising costs of healthcare. However, as Al technologies 
have rapidly moved from controlled lab environments into real-life health contexts, new social 
concerns are also fast emerging. 

The Expanding Scale and Scope of Algorithmic Health Infrastructures 

Advances in machine learning techniques and cloud-computing resources have made it possible 
to classify and analyze large amounts of medical data, allowing the automated and accurate 
detection of conditions like diabetic retinopathy and forms of skin cáncer in medical settings. 428 
At the same time, eager to apply Al techniques to health challenges, technology companies have 
been analyzing everyday experiences like going for a walk, food shopping, sleeping, and 
menstruating to make inferences and predictions about people's health behavior and status 429 

While such developments may offer future positive health benefits, little empirical research has 
been published about how Al will impact patient health outcomes or experiences of care. 
Furthermore, the data- and cloud-computing resources required fortraining models to Al health 
systems have created troubling new opportunities, expanding what counts as "health data,” but 
also the boundaries of healthcare. The scope and scale of these new "algorithmic health 
infrastructures" 430 give rise to a number of social, economic, and political concerns. 

The proliferation of corporate-clinical alliances for sharing data to train Al models illustrates these 
infrastructural impacts. The resulting commercial incentives and conflicts of interest have made 
ethical and legal issues around health data front-page news. Most recently, a whistle-blower 
report alerted the public to serious privacy risks stemming from a partnership, known as Project 
Nightingale, between Google and Ascensión, 431 one of the largest nonprofit health systems in the 
US. The report claimed that patient data transferred between Ascensión and Google was not 
"de-identified." 432 Google helped migrate Ascension's infrastructure to their cloud environment, and 
in return received access to hundreds of thousands of privacy-protected patient medical records 
to use in developing Al Solutions for Ascensión and also to sell to other healthcare systems. 433 

Google, however, is not alone. Microsoft, IBM, Apple, Amazon, and Facebook, as well as a wide 
range of healthcare start-ups, have all made lucrative "data partnership" agreements with a wide 
range of healthcare organizations (including many university research hospitals and insurance 
companies) to gain access to health data for the training and development of Al-driven health 
systems. 434 Several of these have resulted in federal probes and lawsuits around improper use of 
patient data. 435 
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However, even when current regulatory policies like HIPAA are strictly followed, security and 
privacy vulnerabilities can exist within largertechnology infrastructures, presenting serious 
challenges for the safe collection and use of Electronic Health Record (EHR) data. New research 
shows that ¡t is possible to accurately link two different de-identified EHR dataseis using 
computational methods, so as to create a more complete history of a patient without using any 
personal health Information of the patient in question 436 Another recent research study showed 
that it is possible to create reconstructions of patients' faces using de-identified MRI images, 
which could then be identified using facial-recognition systems. 437 Similar concerns have 
prompted a lawsuit against the University of Chicago Medical Center and Google claiming that 
Google is "uniquely able to determine the identity of almost every medical record the university 
released" due to its expertise and resources in Al development. 438 The potential harm from 
misuse of these new health data capabilities is of grave concern, especially as Al health 
technologies continué to focus on predicting risks that could impact healthcare access or 
stigmatize individuáis, such as recent attempts to diagnose complex behavioral health conditions 
like depression and schizophrenia from social-media data 439 

New Social Challenges for the Healthcare Community 

This year a number of reports, papers, and op-eds were published on Al ethics in healthcare. 440 
Although mostly generated by physicians and medical ethicists in Europe and North America, 
these early efforts are important for better understanding the situated uses of Al systems in 
healthcare. 

For example, the European and North American Radiology Societies recently issued a statement 
that outlines key ethical issues for the field, including algorithmic and automation bias in relation 
to medical imaging. 441 Radiology is currently one of the medical specialties where Al systems are 
the most advanced. The statement openly acknowledges how clinicians are reckoning with the 
increased valué and potential harms around health data used for Al systems: "Al has noticeably 
altered our perception of radiology data-their valué, how to use them, and how they may be 
misused." 442 

These challenges inelude possible social harms for patients, such as the potential for clinical 
decisions to be nudged or guided by Al systems in ways that dorít (necessarily) bring people 
health benefits, but are in Service to quality metric requirements or increased profit. Importantly, 
misuses also extend beyond the ethics of patient care to consider how Al technologies are 
reshaping medical organizations themselves (e.g., "radiologist and radiology departments will 
also be data" for healthcare administrators) 443 and the wider health domain by "blurring the line" 
between academic research and commercial Al uses of health data. 444 

Importantly, medical groups are also pushing back against the techno-solutionist promises of Al, 
crafting policy recommendations to address social concerns. For example, the Academy of 
Medical Royal Colleges (UK) 2019 report, "Artificial Intelligence in Healthcare," pragmatically 
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States: "Politicians and policymakers should avoid thinking that Al is going to solve all the 
problems the health and care systems across the UK are facing." 445 The American Medical 
Association has been working on an Al agenda for healthcare, too, also adopting the policy 
"Augmented Intelligence in Health Care" 446 as a framework for thinking about Al in relation to 
múltiple stakeholder concerns, which inelude the needs of physicians, patients, and the broader 
healthcare community. 

There have also been recent calis for setting a more engaged agenda around Al and health. This 
year Eric Topol, a physician and AI/ML researcher, questioned the promises of Al to fix systemic 
healthcare issues, like clinician burnout, without the collective action and involvement of 
healthcare workers. 447 Physician organizing is needed not because doctors should fear being 
replaced by Al, but to ensure that Al benefits people's experiences of care. "The potential of A.l. to 
restore the human dimensión in health care,” Topol argües, "will depend on doctors stepping up to 
make their voices heard." 448 

More voices are urgently needed at the table—including the expertise of patient groups, family 
caregivers, community health workers, and nurses-in orderto better understand how Al 
technologies will impact diverse populations and health contexts. We have seen how overly 
narrow approaches to Al in health have resulted in systems that failed to account for darker skin 
tones in medical imaging data, 449 and cáncer treatment recommendations that could lead to 
racially disparate outeomes due to training data from predominantly white patients. 450 

Importantly, algorithmic bias in health data cannot always be corrected by gathering more data, 
but requires understanding the social context of the health data that has already been collected. 
Recently, Optum’s algorithm designed to identify "high-risk" patients in the US was based on the 
number of medical Services a person used, but didn't account for the numerous socioeconomic 
reasons around the nonuse of needed health Services, such as being underinsured orthe inability 
to take time off from work. 451 With long histories of addressing such social complexities, research 
from fields like medical sociology and anthropology, nursing, human-computer interaction, and 
public health is needed to protect against the implementation of Al systems that (even when 
designed with good intentions) worsen health inequities 452 


2.7 Advances in the Machine Leaming Community 

The Tough Road Toward Sociotechnical Perspectives 

As research and perspectives on the social implications of Al evolve, machine learning (ML) 
research communities are realizing the limitations of narrow "fairness” definitions and are shifting 
their focus to more impactful interventions and strategies, as well as fostering an increased 
openness toward active inclusión and engagement with other disciplines. 
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ln our 2018 Al Now Report, we critically assessed the affordances and limitations of technical 
fixes to problems of fairness. 453 Since then, several convincing critiques have emerged that 
further explain how these approaches fundamentally distract from more urgent issues, 454 abstract 
away societal context 455 are incommensurate with the political reality of how data scientists 
approach "problem formulation," 456 and fail to address the hierarchical logic that produces 
unlawful discrimination. 457 

Responding to these criticisms, many technical researchers have turned to the use of so-called 
"causar or "counterfactual" fairness methods. Ratherthan relying on the correlations that most 
ML models use to make their predictions, these approaches aim to draw causal diagrams that 
explain how differenttypes of data produce various outcomes. When analyzed for use of 
sensitive or protected categories, such as race or gender, these researchers seekto declare an 
ML "fair" if factors like race or gender do not causally ¡nfluence the model's prediction. 

While the intentions behind this work may be commendable, there are still olear limitations to 
these approaches, primarily in their abilityto address historical disparities and ongoing structural 
injustices 458 As Lily Hu explains in the context of racial health disparities, "Whatever [level of] 
health Black people would have had in some convoluted counterfactual scenario is frankly 
irrelevant to the question of whether actually existing inequality is a matter of injustice—let alone 
what can be doneto remedy it." 459 In addition, the valué of these assessments hinges on how to 
define which individual characteristics should or should not factor into the algorithm's final 
prediction 460 Such decisions are often themselves politically, culturally, and socially influenced, 
and the power imbalance between those making such determinations and those impacted 
remains olear and unaddressed. 461 

Techniques for interpreting and explaining ML systems have also gained popularity. However, they 
suffer from many of these same critiques, and have been shown to be fundamentally fragile and 
prone to manipulation, 462 and to ignore a long history of insights from the social Sciences. 463 

As a result, some researchers have begun to push harder on the need for interdisciplinary 
approaches, 464 and for integrating lessons from social Sciences and humanities into the practice 
of developing Al systems 465 Some practical strategies have emerged, including methods to 
document the development of machine learning models to enforce some level of additional 
ethical reflection and reporting throughout the engineering process 466 Industry-led efforts by the 
Partnership on Al and IEEE are also attempting to consolídate these documentation proposals 
and to standardize reporting requirements across the ¡ndustry. 467 

This year, more algorithmic audits also uncovered disproportionate performance or biases within 
Al systems ranging from self-driving-car software that performed differently for darker- and 
lighter-skinned pedestrians, 468 gender bias in online biographies, 469 skewed representations in 
object recognition from lower-income environments, 470 racial differences in algorithmic pricing, 471 
and differential prioritization in healthcare, 472 as well as performance disparities in facial 
recognition 473 In several cases, these audits had a tangible impact on improving the lives of 
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people unfairly affected. 474 They also had a substantial impact on policy discussions 475 For 
instance, two audit studies of facial-recognition systems, including the widely recognized Gender 
Shades 476 led to subsequent audit studies by the National Institute of Standards and 
Technology 477 and other researchers, 478 including the ACLU of Northern California's audits of 
Amazon Rekognition, which falsely matched 28 Congress members 479 and 27 mostly minority 
athletes to criminal mugshots. 480 

Confronting Al's Inherent Vulnerabilities 

Concerns over the vulnerabilities of Al systems also gained greater attention this year, highlighting 
the urgent need for them to be subjected to the same scrutiny applied to automation 
technologies in other engineering fields, such as aviation and power systems. 

Among the most urgent vulnerabilities to address is the danger of data-poisoning techniques, a 
method of exploitation in which a bad actor can fiddle with Al training data to alter a system's 
decisions. A classic example is spam filtering, where intentionally curating the contení of 
messages that teach a spam filter how spam looks can help certain types of spam pass through 
the filter undetected 481 

A second type of Al vulnerability that can be exploited is the so-called "back door," which lets 
attackers find ways to infíltrate an Al system through code that malicious programmers embed in 
systems they trained or designed for later infiltration by a bad actor. Researchers at NYU showed 
that back-door attacks may result in a model that has state-of-the-art performance on the user's 
training and validation samples (dataseis used to test Al models), but behaves badly when 
confronted with specific attacker-chosen inputs 482 The researchers used the back door to poison 
an Al road sign detector (commonly used in autonomous vehicles) into misclassifying US stop 
signs. And when they "retrained" the model to work on Swedish stop signs, the earlier poisoning 
effects carried over. This type of vulnerability raises serious concerns given the rapid move 
toward outsourcing the training procedures of ML models to cloud platforms. 483 

A related trend is the move to reduce training costs by repurposing and retraining Al models for 
new or specific tasks, a phenomenon called transfer learning. Transfer learning is particularly 
popular for applications that require large models, such as natural-language Processing 484 or 
image classification. 485 Instead of starting from scratch, one retrains the parameters of a 
preexisting central model with more specific data for a new task or domain. Researchers show 
that this "centralization of model training increases their vulnerability to misclassification attacks,” 
especially when such central models are publicly available. 486 

Adversarial attacks are particularly effective against systems with a high number of inputs, which 
are the variables that an Al model considers to make a decisión or prediction when deployed. 487 
This reliance on a large number of inputs is inherent to computer-vision systems, where typically 
each pixel is an input. It is likely also an issue for applications where automated decisión systems 
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rely on a variety of inputs to make predictions about human behavior or preferences. Such 
models rely on diverse data sources, including social-network data, search entries, location 
tracking, energy use, and other revealing data about individual behavior and preferences. Such 
vulnerabilities expose people to misclassification, hacking, and strategic manipulation. 
Researchers from Harvard and MIT convincingly explained these concerns for the context of 
medical diagnostics 488 

While research exposing technical vulnerabilities and proposing new defenses against them is 
now of high priority, building robust machine learning systems is still an elusive goal. A group of 
researchers across Google Brain, MIT, and the University of Tübingen recently surveyed the field 
and concluded that few defense mechanisms have succeeded. There is consensus in the field 
that most papers that propose defenses are quickly shown to be either incorrect or insufficient. 
The group observes that "[r]esearchers must be very careful to not deceive themselves 
unintentionally when performing evaluations." 489 

We must be extra careful when bringing Al systems to contexts where their errors lead to social 
harm. Similar to our discussion of fairness and bias in the 2018 Al Now report, 490 any debate 
about vulnerabilities should approach issues of power and hierarchy, looking at who is in a 
position to produce and profit from these systems, who determines how vulnerabilities are 
accounted for and addressed, and who is most likely to be harmed. 

Despite the fact that social Sciences and humanities approaches have a long history in 
information security and risk management, 491 research that addresses both social and technical 
dimensions in security is necessary, but still relatively nascent. 492 Central in this challenge is 
redrawing the boundaries of analysis and design to expand beyond the algorithm, 493 and securing 
channels for all affected stakeholders to democratically steer system development and to dissent 
when concerns arise. 494 


CONCLUSION 


Despite the growth of ethical frameworks, Al systems continué to be deployed rapidly across 
domains of considerable social significance-in healthcare, education, employment, criminal 
justice, and many others-without appropriate safeguards or accountability structures in place. 
Many urgent concerns remain, and the agenda of issues to be addressed continúes to grow: the 
environmental harms caused by Al systems are considerable, from extraction of materials from 
our earth to the extraction of labor from our communities. In healthcare, increasing dependence 
on Al systems will have life-or-death consequences. New research also highlights how Al systems 
are particularly prone to security vulnerabilities and how the companies building these systems 
are inciting fundamental changes to the landscape of our communities, resulting in geographic 
displacement. 
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Yet the movements of the past year give reason to hope, marked by a groundswell of pushback 
from both expected and unexpected places, from regulators and researchers to community 
organizers and activists to workers and advocates. Together, they are building new coalitions 
upon legacies of older ones, and forging new bonds of solidarity. If the past year has shown us 
anything, it is that our future will not be determined by the inevitable progress of Al, ñor are we 
doomed to a dystopic future. The implications of Al will be determined by us—and there is much 
work ahead to ensure that the future looks bright. 
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